2020-08-24T00:33:18.965 INFO:root:teuthology version: 1.0.0-c0ecc52 2020-08-24T00:33:18.968 DEBUG:teuthology.run:Teuthology command: teuthology --block --lock --owner scheduled_teuthology@teuthology --verbose --archive /home/teuthworker/archive/teuthology-2020-08-19_03:30:03-rados-octopus-distro-basic-smithi/5358535 --name teuthology-2020-08-19_03:30:03-rados-octopus-distro-basic-smithi --description rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} -- /tmp/teuthology-worker.7nbyy4y_.tmp 2020-08-24T00:33:18.994 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-24T00:33:19.058 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2020-08-19_03:30:03-rados-octopus-distro-basic-smithi/5358535 branch: octopus description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/osd} email: ceph-qa@ceph.io first_in_suite: false job_id: '5358535' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2020-08-19_03:30:03-rados-octopus-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '18.04' overrides: admin_socket: branch: octopus ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0a941e55043ccee6d35a8717bec55ff7581e67d8 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 0a941e55043ccee6d35a8717bec55ff7581e67d8 workunit: branch: octopus sha1: 0a941e55043ccee6d35a8717bec55ff7581e67d8 owner: scheduled_teuthology@teuthology priority: 1000 repo: git://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 0a941e55043ccee6d35a8717bec55ff7581e67d8 sleep_before_teardown: 0 suite: rados suite_branch: octopus suite_path: /home/teuthworker/src/git.ceph.com_ceph_octopus/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph.git suite_sha1: 0a941e55043ccee6d35a8717bec55ff7581e67d8 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master timestamp: 2020-08-19_03:30:03 tube: smithi user: teuthology verbose: true worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.29144 2020-08-24T00:33:19.059 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_octopus/qa; will attempt to use it 2020-08-24T00:33:19.060 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_octopus/qa/tasks 2020-08-24T00:33:19.061 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-08-24T00:33:19.063 INFO:teuthology.task.internal:Checking packages... 2020-08-24T00:33:19.107 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '0a941e55043ccee6d35a8717bec55ff7581e67d8' 2020-08-24T00:33:19.108 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-08-24T00:33:19.109 INFO:teuthology.packaging:ref: None 2020-08-24T00:33:19.109 INFO:teuthology.packaging:tag: None 2020-08-24T00:33:19.110 INFO:teuthology.packaging:branch: octopus 2020-08-24T00:33:19.110 INFO:teuthology.packaging:sha1: 0a941e55043ccee6d35a8717bec55ff7581e67d8 2020-08-24T00:33:19.111 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2020-08-24T00:33:19.531 INFO:teuthology.task.internal:Found packages for ceph version 15.2.4-705-g5943365-1bionic 2020-08-24T00:33:19.532 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-08-24T00:33:19.544 INFO:teuthology.task.internal:no buildpackages task found 2020-08-24T00:33:19.544 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-08-24T00:33:20.294 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-08-24T00:33:20.295 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-24T00:33:20.428 DEBUG:teuthology.lock.ops:lock_many request: {'locked_by': 'scheduled_teuthology@teuthology', 'count': 1, 'machine_type': 'smithi', 'description': '/home/teuthworker/archive/teuthology-2020-08-19_03:30:03-rados-octopus-distro-basic-smithi/5358535'} 2020-08-24T00:33:20.448 DEBUG:teuthology.lock.ops:locked ubuntu@smithi060.front.sepia.ceph.com 2020-08-24T00:33:20.491 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f90c94378>, signals=[15]) 2020-08-24T00:33:20.778 INFO:teuthology.provision.fog.smithi060:Scheduling deploy of ubuntu 18.04 2020-08-24T00:33:21.452 INFO:teuthology.orchestra.console:Power off smithi060 2020-08-24T00:33:21.454 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-08-24T00:33:21.488 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-24T00:33:25.522 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-24T00:33:29.644 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-24T00:33:29.860 INFO:teuthology.orchestra.console:Power off for smithi060 completed 2020-08-24T00:33:29.962 INFO:teuthology.orchestra.console:Power on smithi060 2020-08-24T00:33:29.963 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-08-24T00:33:29.994 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-24T00:33:34.026 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi060.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-08-24T00:33:34.250 INFO:teuthology.orchestra.console:Power on for smithi060 completed 2020-08-24T00:33:34.352 INFO:teuthology.provision.fog.smithi060:Waiting for deploy to finish 2020-08-24T00:36:21.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:36:30.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:36:39.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:36:48.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:36:57.396 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:37:37.505 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:37:37.523 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-08-24T00:37:43.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:37:43.545 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-08-24T00:37:49.550 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:37:52.583 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2020-08-24T00:37:58.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:37:58.679 INFO:teuthology.orchestra.run:Running command with timeout 600 2020-08-24T00:37:58.682 INFO:teuthology.orchestra.run.smithi060:> while [ ! -e '/.cephlab_net_configured' ]; do sleep 5; done 2020-08-24T00:37:59.107 INFO:teuthology.provision.fog.smithi060:Node is ready 2020-08-24T00:37:59.108 INFO:teuthology.orchestra.run.smithi060:> hostname 2020-08-24T00:37:59.155 INFO:teuthology.orchestra.run.smithi060.stdout:smithi060.front.sepia.ceph.com 2020-08-24T00:37:59.156 INFO:teuthology.orchestra.run.smithi060:> grep smithi060.front.sepia.ceph.com /etc/hosts 2020-08-24T00:37:59.204 INFO:teuthology.orchestra.run.smithi060.stdout:172.21.15.60 smithi060.front.sepia.ceph.com smithi060 2020-08-24T00:37:59.205 INFO:teuthology.orchestra.run.smithi060:> sudo hostname smithi060 2020-08-24T00:37:59.267 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/smithi060.front.sepia.ceph.com/smithi060/g' /etc/hosts 2020-08-24T00:37:59.325 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/172.21.15.60/172.21.15.60/g' /etc/hosts 2020-08-24T00:37:59.380 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i -e 's/smithi060.front.sepia.ceph.com/smithi060/g' /etc/hostname 2020-08-24T00:37:59.437 INFO:teuthology.orchestra.run.smithi060:> sudo hostname smithi060 2020-08-24T00:37:59.494 INFO:teuthology.provision.fog.smithi060:Deploy complete! 2020-08-24T00:37:59.495 DEBUG:teuthology.parallel:result is None 2020-08-24T00:37:59.622 INFO:teuthology.lock.ops:Checking smithi060.front.sepia.ceph.com 2020-08-24T00:37:59.624 INFO:teuthology.lock.ops:New key found. Updating... 2020-08-24T00:37:59.643 INFO:teuthology.lock.ops:Updating [smithi060.front.sepia.ceph.com]: set os type and version on server 2020-08-24T00:37:59.644 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-08-24T00:37:59.645 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:37:59.766 INFO:teuthology.orchestra.run.smithi060:> true 2020-08-24T00:38:00.023 INFO:teuthology.orchestra.run.smithi060:> uname -m 2020-08-24T00:38:00.075 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2020-08-24T00:38:00.076 INFO:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2020-08-24T00:38:00.122 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2020-08-24T00:38:00.123 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-08-24T00:38:00.123 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2020-08-24T00:38:00.123 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2020-08-24T00:38:00.123 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-08-24T00:38:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="18.04" 2020-08-24T00:38:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2020-08-24T00:38:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-08-24T00:38:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-08-24T00:38:00.124 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-08-24T00:38:00.125 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=bionic 2020-08-24T00:38:00.125 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=bionic 2020-08-24T00:38:00.126 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2020-08-24T00:38:00.156 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-08-24T00:38:00.175 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi060.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCk/RYrbUBlldTLne6pTfbL/y7LeoDtW9aq6v+XowGN3wsA/R1KoxXd3OBw8RI+l7wgbhqjb/3ccih/sLsh5+yQyQEi8hY5nrQbGCrCF1bowQwnj/dSwdPxdcfQ6PZQw5j/Q9jukwiWo5JRdGxvRZH2pAjyjBbLMyE6LjiO34GFlkdTgrmqRNTuDVP0kCnEPwLaww8yOyhiV049k36hG34UWZIqHMxXJ8hJv52e6iWl+x3pejovVYw2Pyew6NYk99sBO1Lhi5WpxS/kUiSHQfl6tkEeupcS2jTniQcegmC1mK3MlQuwp0gvW5uXGnRr91tO18GqjUmjy15mga5BEWId 2020-08-24T00:38:00.176 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-08-24T00:38:00.308 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-08-24T00:38:00.316 INFO:teuthology.task.internal:Saving configuration 2020-08-24T00:38:00.332 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-08-24T00:38:00.612 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-08-24T00:38:00.631 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2020-08-24 00:33:20.435337', 'description': '/home/teuthworker/archive/teuthology-2020-08-19_03:30:03-rados-octopus-distro-basic-smithi/5358535', 'locked_by': 'scheduled_teuthology@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJBx++SblJWuQYQbKHFUbmUtDdtEDHLFr4ZzXUbGE+9E', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi060.front.sepia.ceph.com'} 2020-08-24T00:38:00.632 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-08-24T00:38:00.821 INFO:teuthology.task.internal:roles: ubuntu@smithi060.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-08-24T00:38:00.822 INFO:teuthology.run_tasks:Running task console_log... 2020-08-24T00:38:00.950 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f90c1f048>, signals=[15]) 2020-08-24T00:38:00.951 INFO:teuthology.run_tasks:Running task internal.connect... 2020-08-24T00:38:01.062 INFO:teuthology.task.internal:Opening connections... 2020-08-24T00:38:01.063 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi060.front.sepia.ceph.com 2020-08-24T00:38:01.065 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:38:01.158 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-08-24T00:38:01.643 INFO:teuthology.orchestra.run.smithi060:> uname -m 2020-08-24T00:38:01.656 INFO:teuthology.orchestra.run.smithi060.stdout:x86_64 2020-08-24T00:38:01.657 INFO:teuthology.orchestra.run.smithi060:> cat /etc/os-release 2020-08-24T00:38:01.707 INFO:teuthology.orchestra.run.smithi060.stdout:NAME="Ubuntu" 2020-08-24T00:38:01.708 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2020-08-24T00:38:01.708 INFO:teuthology.orchestra.run.smithi060.stdout:ID=ubuntu 2020-08-24T00:38:01.708 INFO:teuthology.orchestra.run.smithi060.stdout:ID_LIKE=debian 2020-08-24T00:38:01.708 INFO:teuthology.orchestra.run.smithi060.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2020-08-24T00:38:01.709 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_ID="18.04" 2020-08-24T00:38:01.709 INFO:teuthology.orchestra.run.smithi060.stdout:HOME_URL="https://www.ubuntu.com/" 2020-08-24T00:38:01.709 INFO:teuthology.orchestra.run.smithi060.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2020-08-24T00:38:01.709 INFO:teuthology.orchestra.run.smithi060.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2020-08-24T00:38:01.709 INFO:teuthology.orchestra.run.smithi060.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2020-08-24T00:38:01.710 INFO:teuthology.orchestra.run.smithi060.stdout:VERSION_CODENAME=bionic 2020-08-24T00:38:01.710 INFO:teuthology.orchestra.run.smithi060.stdout:UBUNTU_CODENAME=bionic 2020-08-24T00:38:01.710 INFO:teuthology.lock.ops:Updating smithi060.front.sepia.ceph.com on lock server 2020-08-24T00:38:01.733 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-08-24T00:38:01.810 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-08-24T00:38:02.270 INFO:teuthology.task.internal:Checking for old test directory... 2020-08-24T00:38:02.271 INFO:teuthology.orchestra.run.smithi060:> test '!' -e /home/ubuntu/cephtest 2020-08-24T00:38:02.282 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-08-24T00:38:02.587 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-08-24T00:38:02.588 INFO:teuthology.orchestra.run.smithi060:> test -z $(ls -A /var/lib/ceph) 2020-08-24T00:38:02.601 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-08-24T00:38:03.452 INFO:teuthology.run_tasks:Running task kernel... 2020-08-24T00:38:04.588 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}} 2020-08-24T00:38:04.589 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2020-08-24T00:38:04.596 INFO:teuthology.orchestra.run.smithi060:> uname -r 2020-08-24T00:38:04.603 INFO:teuthology.orchestra.run.smithi060.stdout:4.15.0-112-generic 2020-08-24T00:38:04.604 INFO:teuthology.task.kernel:Running kernel on smithi060: 4.15.0-112-generic 2020-08-24T00:38:04.604 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get clean 2020-08-24T00:38:04.707 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-08-24T00:38:04.977 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-08-24T00:38:04.977 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2020-08-24T00:38:05.039 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2020-08-24T00:38:05.314 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2020-08-24T00:38:05.734 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 252 kB in 1s (314 kB/s) 2020-08-24T00:38:07.494 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-08-24T00:38:07.543 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2020-08-24T00:38:07.635 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-08-24T00:38:07.887 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-08-24T00:38:07.887 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-08-24T00:38:07.977 INFO:teuthology.orchestra.run.smithi060.stderr:E: Unable to locate package linux-image-current-generic 2020-08-24T00:38:07.978 DEBUG:teuthology.orchestra.run:got remote process result: 100 2020-08-24T00:38:07.979 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2020-08-24T00:38:08.057 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-08-24T00:38:08.292 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-08-24T00:38:08.293 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-08-24T00:38:08.510 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic is already the newest version (4.15.0.112.100). 2020-08-24T00:38:08.510 INFO:teuthology.orchestra.run.smithi060.stdout:linux-image-generic set to manually installed. 2020-08-24T00:38:08.510 INFO:teuthology.orchestra.run.smithi060.stdout:0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. 2020-08-24T00:38:08.512 INFO:teuthology.orchestra.run.smithi060:> dpkg -s linux-image-generic 2020-08-24T00:38:08.530 INFO:teuthology.orchestra.run.smithi060.stdout:Package: linux-image-generic 2020-08-24T00:38:08.531 INFO:teuthology.orchestra.run.smithi060.stdout:Status: install ok installed 2020-08-24T00:38:08.532 INFO:teuthology.orchestra.run.smithi060.stdout:Priority: optional 2020-08-24T00:38:08.532 INFO:teuthology.orchestra.run.smithi060.stdout:Section: kernel 2020-08-24T00:38:08.532 INFO:teuthology.orchestra.run.smithi060.stdout:Installed-Size: 16 2020-08-24T00:38:08.532 INFO:teuthology.orchestra.run.smithi060.stdout:Maintainer: Ubuntu Kernel Team 2020-08-24T00:38:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:Architecture: amd64 2020-08-24T00:38:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:Source: linux-meta 2020-08-24T00:38:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:Version: 4.15.0.112.100 2020-08-24T00:38:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:Provides: wireguard-modules (= 1.0.20200611-1ubuntu1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.9) 2020-08-24T00:38:08.533 INFO:teuthology.orchestra.run.smithi060.stdout:Depends: linux-image-4.15.0-112-generic, linux-modules-extra-4.15.0-112-generic, linux-firmware, intel-microcode, amd64-microcode 2020-08-24T00:38:08.534 INFO:teuthology.orchestra.run.smithi060.stdout:Recommends: thermald 2020-08-24T00:38:08.534 INFO:teuthology.orchestra.run.smithi060.stdout:Description: Generic Linux kernel image 2020-08-24T00:38:08.534 INFO:teuthology.orchestra.run.smithi060.stdout: This package will always depend on the latest generic kernel image 2020-08-24T00:38:08.534 INFO:teuthology.orchestra.run.smithi060.stdout: available. 2020-08-24T00:38:08.535 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2020-08-24T00:38:08.535 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-08-24T00:38:08.535 INFO:teuthology.orchestra.run.smithi060:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-08-24T00:38:08.608 INFO:teuthology.orchestra.run.smithi060.stdout:ttyS1 2020-08-24T00:38:08.635 INFO:teuthology.run_tasks:Running task internal.base... 2020-08-24T00:38:08.658 INFO:teuthology.task.internal:Creating test directory... 2020-08-24T00:38:08.659 INFO:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-08-24T00:38:08.688 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-08-24T00:38:09.206 INFO:teuthology.run_tasks:Running task internal.archive... 2020-08-24T00:38:10.123 INFO:teuthology.task.internal:Creating archive directory... 2020-08-24T00:38:10.124 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-08-24T00:38:10.146 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-08-24T00:38:12.405 INFO:teuthology.task.internal:Enabling coredump saving... 2020-08-24T00:38:15.173 INFO:teuthology.orchestra.run.smithi060:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-08-24T00:38:15.202 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:38:15.212 INFO:teuthology.orchestra.run.smithi060.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:38:15.213 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-08-24T00:38:15.222 INFO:teuthology.task.internal:Configuring sudo... 2020-08-24T00:38:15.222 INFO:teuthology.orchestra.run.smithi060:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-08-24T00:38:15.272 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-08-24T00:38:15.772 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-08-24T00:38:15.783 INFO:teuthology.orchestra.run.smithi060:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-08-24T00:38:15.792 INFO:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-08-24T00:38:15.839 INFO:teuthology.orchestra.run.smithi060:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-08-24T00:38:15.887 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-08-24T00:38:15.976 INFO:teuthology.orchestra.run.smithi060:> sudo service rsyslog restart 2020-08-24T00:38:16.059 INFO:teuthology.run_tasks:Running task internal.timer... 2020-08-24T00:38:16.989 INFO:teuthology.task.internal:Starting timer... 2020-08-24T00:38:16.989 INFO:teuthology.run_tasks:Running task pcp... 2020-08-24T00:38:18.227 INFO:teuthology.run_tasks:Running task selinux... 2020-08-24T00:38:18.797 DEBUG:teuthology.task.selinux:Excluding smithi060: OS 'ubuntu' does not support SELinux 2020-08-24T00:38:18.798 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-08-24T00:38:18.798 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2020-08-24T00:38:18.798 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-08-24T00:38:18.799 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-08-24T00:38:19.003 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated; assuming it is current 2020-08-24T00:38:19.004 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-08-24T00:38:19.052 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-08-24T00:38:19.054 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi060.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-08-24T00:38:19.451 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-08-24T00:38:20.525 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-08-24T00:38:22.990 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-08-24T00:38:23.012 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:23 +0000 (0:00:00.076) 0:00:00.076 ********* 2020-08-24T00:38:24.854 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi060.front.sepia.ceph.com] 2020-08-24T00:38:24.935 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:24 +0000 (0:00:01.923) 0:00:01.999 ********* 2020-08-24T00:38:25.362 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-08-24T00:38:25.403 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:25 +0000 (0:00:00.466) 0:00:02.466 ********* 2020-08-24T00:38:26.058 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** changed: [smithi060.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-08-24T00:38:26.098 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:26 +0000 (0:00:00.692) 0:00:03.158 ********* 2020-08-24T00:38:26.599 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.071658", "end": "2020-08-24 00:38:26.557130", "rc": 0, "start": "2020-08-24 00:38:26.485472", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2020-08-24T00:38:26.638 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:26 +0000 (0:00:00.543) 0:00:03.702 ********* 2020-08-24T00:38:27.050 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-08-24T00:38:27.088 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:27 +0000 (0:00:00.451) 0:00:04.153 ********* 2020-08-24T00:38:27.994 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1598229507.1075485-250888994867830/source", "state": "file", "uid": 0} 2020-08-24T00:38:28.037 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:28 +0000 (0:00:00.948) 0:00:05.101 ********* 2020-08-24T00:38:28.502 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-08-24T00:38:28.504 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-08-24T00:38:28.506 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-08-24T00:38:28.699 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** 2020-08-24T00:38:28.701 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-08-24T00:38:28.708 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-08-24T00:38:28.856 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:28 +0000 (0:00:00.819) 0:00:05.921 ********* 2020-08-24T00:38:29.630 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi060.front.sepia.ceph.com] 2020-08-24T00:38:29.717 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-08-24T00:38:29.718 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:29 +0000 (0:00:00.860) 0:00:06.781 ********* 2020-08-24T00:38:30.181 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi060.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.196962", "end": "2020-08-24 00:38:30.143448", "rc": 0, "start": "2020-08-24 00:38:29.946486", "stderr": "", "stderr_lines": [], "stdout": "f2a941ba8b53bcd41d41bd9c6c0efbccae5fc796", "stdout_lines": ["f2a941ba8b53bcd41d41bd9c6c0efbccae5fc796"]} 2020-08-24T00:38:30.221 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.504) 0:00:07.286 ********* 2020-08-24T00:38:30.381 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-08-24T00:38:30.383 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.008632", "end": "2020-08-24 00:38:30.341098", "failed_when_result": false, "rc": 0, "start": "2020-08-24 00:38:30.332466", "stderr": "", "stderr_lines": [], "stdout": "f2a941ba8b53bcd41d41bd9c6c0efbccae5fc796", "stdout_lines": ["f2a941ba8b53bcd41d41bd9c6c0efbccae5fc796"]} 2020-08-24T00:38:30.428 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.207) 0:00:07.493 ********* 2020-08-24T00:38:30.466 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-08-24T00:38:30.468 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-08-24T00:38:30.512 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.081) 0:00:07.574 ********* 2020-08-24T00:38:30.540 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-08-24T00:38:30.542 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.578 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.068) 0:00:07.643 ********* 2020-08-24T00:38:30.623 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.662 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.083) 0:00:07.726 ********* 2020-08-24T00:38:30.694 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-08-24T00:38:30.696 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.736 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.073) 0:00:07.800 ********* 2020-08-24T00:38:30.766 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-08-24T00:38:30.768 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.804 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.068) 0:00:07.869 ********* 2020-08-24T00:38:30.835 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-08-24T00:38:30.837 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.874 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:30 +0000 (0:00:00.070) 0:00:07.939 ********* 2020-08-24T00:38:30.925 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-08-24T00:38:30.927 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.937 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.938 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.948 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.950 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.959 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.961 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.971 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratzas@laptop 65tzczvq6U9MaLuTtZUWJA 36cc4b5af314d8d2ae1cc974086b62fd4d9a722b1469f7cf00ba647241b834c9"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.982 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.993 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:30.995 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.022 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.027 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.040 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "ite 2020-08-24T00:38:31.042 INFO:teuthology.task.ansible.out:m", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.050 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.052 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.062 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.063 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zyan', 'ovpn': 'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.073 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.075 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.085 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.097 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.107 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.119 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.130 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.132 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.142 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.145 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.146 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.157 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.159 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.169 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.171 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.181 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.182 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.193 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.204 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.205 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.215 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.217 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.238 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.249 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.251 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.261 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.264 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.273 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.275 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.285 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.286 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.297 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.298 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.308 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.308 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.322 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.324 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.334 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.336 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.351 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.362 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.373 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.384 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.398 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.410 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.424 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.430 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.432 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.442 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.464 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomo 2020-08-24T00:38:31.468 INFO:teuthology.task.ansible.out:n@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.476 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.486 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.497 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.502 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.510 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.527 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.529 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.539 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.541 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.550 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.561 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.563 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.573 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.574 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.584 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.595 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.597 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.607 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.645 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.676 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ff 2020-08-24T00:38:31.682 INFO:teuthology.task.ansible.out:ddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.689 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.712 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.724 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.731 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.751 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.753 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.817 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:31 +0000 (0:00:00.936) 0:00:08.876 ********* 2020-08-24T00:38:31.854 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-08-24T00:38:31.890 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:31 +0000 (0:00:00.079) 0:00:08.955 ********* 2020-08-24T00:38:31.892 INFO:teuthology.task.ansible.out: 2020-08-24T00:38:31.923 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2020-08-24T00:38:31.925 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:31.965 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:31 +0000 (0:00:00.074) 0:00:09.030 ********* 2020-08-24T00:38:32.002 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2020-08-24T00:38:32.004 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.042 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:32 +0000 (0:00:00.076) 0:00:09.106 ********* 2020-08-24T00:38:32.086 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.121 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:32 +0000 (0:00:00.078) 0:00:09.185 ********* 2020-08-24T00:38:32.162 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-08-24T00:38:32.164 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.207 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:32 +0000 (0:00:00.086) 0:00:09.271 ********* 2020-08-24T00:38:32.251 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2020-08-24T00:38:32.252 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.292 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:32 +0000 (0:00:00.085) 0:00:09.357 ********* 2020-08-24T00:38:32.334 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** skipping: [smithi060.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.345 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.364 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.368 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.379 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.379 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.381 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.391 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.392 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.403 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.405 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.415 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.417 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.427 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.427 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.428 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.449 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.455 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.457 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.470 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.479 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.482 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.490 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.501 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.514 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.544 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.546 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.549 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.551 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:32.602 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:32 +0000 (0:00:00.309) 0:00:09.667 ********* 2020-08-24T00:38:33.141 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-08-24T00:38:33.146 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "53e1825fd499fdf6e3f0e53caccfba87eadbf958", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2020-08-24T00:38:33.186 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.581) 0:00:10.248 ********* 2020-08-24T00:38:33.250 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** ok: [smithi060.front.sepia.ceph.com] => { "msg": "Host smithi060.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2020-08-24T00:38:33.297 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.111) 0:00:10.360 ********* 2020-08-24T00:38:33.351 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-08-24T00:38:33.353 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2020-08-24T00:38:33.407 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.111) 0:00:10.472 ********* 2020-08-24T00:38:33.469 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-08-24T00:38:33.471 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:33.506 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.099) 0:00:10.571 ********* 2020-08-24T00:38:33.550 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:33.590 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.080) 0:00:10.652 ********* 2020-08-24T00:38:33.632 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:33.681 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.093) 0:00:10.745 ********* 2020-08-24T00:38:33.714 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-08-24T00:38:33.722 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:33.752 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.071) 0:00:10.816 ********* 2020-08-24T00:38:33.783 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:33.827 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.072) 0:00:10.889 ********* 2020-08-24T00:38:33.854 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-08-24T00:38:33.856 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:33.899 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.074) 0:00:10.963 ********* 2020-08-24T00:38:33.940 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-08-24T00:38:33.942 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:33.982 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:33 +0000 (0:00:00.079) 0:00:11.043 ********* 2020-08-24T00:38:34.009 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-08-24T00:38:34.011 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.050 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.069) 0:00:11.112 ********* 2020-08-24T00:38:34.089 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-08-24T00:38:34.092 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-08-24T00:38:34.104 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-08-24T00:38:34.138 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.089) 0:00:11.202 ********* 2020-08-24T00:38:34.170 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.214 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.069) 0:00:11.272 ********* 2020-08-24T00:38:34.238 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.279 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.070) 0:00:11.342 ********* 2020-08-24T00:38:34.319 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-08-24T00:38:34.326 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-08-24T00:38:34.356 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.078) 0:00:11.421 ********* 2020-08-24T00:38:34.388 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-08-24T00:38:34.390 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.426 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.070) 0:00:11.491 ********* 2020-08-24T00:38:34.457 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-08-24T00:38:34.459 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-08-24T00:38:34.498 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.071) 0:00:11.563 ********* 2020-08-24T00:38:34.531 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-08-24T00:38:34.533 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.578 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.070) 0:00:11.633 ********* 2020-08-24T00:38:34.599 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-08-24T00:38:34.601 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.641 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.071) 0:00:11.705 ********* 2020-08-24T00:38:34.670 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-08-24T00:38:34.672 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.717 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.075) 0:00:11.781 ********* 2020-08-24T00:38:34.750 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-08-24T00:38:34.752 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.789 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.071) 0:00:11.853 ********* 2020-08-24T00:38:34.826 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-08-24T00:38:34.856 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.068) 0:00:11.921 ********* 2020-08-24T00:38:34.908 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:34.960 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:34 +0000 (0:00:00.104) 0:00:12.025 ********* 2020-08-24T00:38:34.991 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-08-24T00:38:34.992 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.029 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.068) 0:00:12.093 ********* 2020-08-24T00:38:35.069 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-08-24T00:38:35.071 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.118 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.083) 0:00:12.177 ********* 2020-08-24T00:38:35.146 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.182 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.069) 0:00:12.246 ********* 2020-08-24T00:38:35.215 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.255 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.071) 0:00:12.317 ********* 2020-08-24T00:38:35.286 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.331 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.077) 0:00:12.395 ********* 2020-08-24T00:38:35.373 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-08-24T00:38:35.411 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.080) 0:00:12.476 ********* 2020-08-24T00:38:35.455 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.499 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.087) 0:00:12.563 ********* 2020-08-24T00:38:35.534 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.570 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.071) 0:00:12.635 ********* 2020-08-24T00:38:35.601 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.645 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.073) 0:00:12.708 ********* 2020-08-24T00:38:35.711 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.713 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.752 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.108) 0:00:12.817 ********* 2020-08-24T00:38:35.794 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-08-24T00:38:35.796 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.832 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.079) 0:00:12.897 ********* 2020-08-24T00:38:35.864 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:35.901 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:35 +0000 (0:00:00.068) 0:00:12.966 ********* 2020-08-24T00:38:36.073 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.009713", "end": "2020-08-24 00:38:36.028165", "rc": 0, "start": "2020-08-24 00:38:36.018452", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-08-24T00:38:36.108 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.207) 0:00:13.173 ********* 2020-08-24T00:38:36.157 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:36.205 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.097) 0:00:13.270 ********* 2020-08-24T00:38:36.237 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-08-24T00:38:36.240 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:36.276 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.069) 0:00:13.340 ********* 2020-08-24T00:38:36.306 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-08-24T00:38:36.308 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:36.353 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.072) 0:00:13.413 ********* 2020-08-24T00:38:36.379 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-08-24T00:38:36.381 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:36.420 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.070) 0:00:13.483 ********* 2020-08-24T00:38:36.449 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:36.491 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.072) 0:00:13.556 ********* 2020-08-24T00:38:36.521 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-08-24T00:38:36.523 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:36.570 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.078) 0:00:13.634 ********* 2020-08-24T00:38:36.600 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-08-24T00:38:36.602 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:36.642 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:36 +0000 (0:00:00.072) 0:00:13.707 ********* 2020-08-24T00:38:37.535 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.747002", "end": "2020-08-24 00:38:37.500366", "rc": 0, "start": "2020-08-24 00:38:36.753364", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:38:37.572 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:37 +0000 (0:00:00.930) 0:00:14.637 ********* 2020-08-24T00:38:37.604 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-08-24T00:38:37.605 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:37.645 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:37 +0000 (0:00:00.072) 0:00:14.709 ********* 2020-08-24T00:38:37.818 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.026142", "end": "2020-08-24 00:38:37.785872", "rc": 0, "start": "2020-08-24 00:38:37.759730", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:38:37.855 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:37 +0000 (0:00:00.210) 0:00:14.920 ********* 2020-08-24T00:38:41.086 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-08-24T00:38:41.088 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:38:41.125 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:41 +0000 (0:00:03.269) 0:00:18.189 ********* 2020-08-24T00:38:42.015 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-08-24T00:38:42.017 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:38:42.054 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:42 +0000 (0:00:00.928) 0:00:19.118 ********* 2020-08-24T00:38:42.091 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-08-24T00:38:42.093 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:42.131 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:42 +0000 (0:00:00.077) 0:00:19.196 ********* 2020-08-24T00:38:42.511 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-08-24T00:38:42.513 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2020-08-24T00:38:42.549 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:42 +0000 (0:00:00.418) 0:00:19.614 ********* 2020-08-24T00:38:43.451 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-08-24T00:38:43.454 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:38:43.497 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:43 +0000 (0:00:00.947) 0:00:20.562 ********* 2020-08-24T00:38:43.871 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-08-24T00:38:43.873 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2020-08-24T00:38:44.240 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2020-08-24T00:38:44.562 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2020-08-24T00:38:44.604 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:44 +0000 (0:00:01.106) 0:00:21.669 ********* 2020-08-24T00:38:44.741 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2020-08-24T00:38:44.780 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:44 +0000 (0:00:00.175) 0:00:21.844 ********* 2020-08-24T00:38:45.150 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-08-24T00:38:45.158 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 12725, "state": "file", "uid": 0} 2020-08-24T00:38:45.482 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2020-08-24T00:38:45.810 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2020-08-24T00:38:45.858 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:45 +0000 (0:00:01.073) 0:00:22.918 ********* 2020-08-24T00:38:45.896 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2020-08-24T00:38:45.940 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:45 +0000 (0:00:00.086) 0:00:23.005 ********* 2020-08-24T00:38:45.993 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-08-24T00:38:45.995 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:46.041 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:46 +0000 (0:00:00.100) 0:00:23.105 ********* 2020-08-24T00:38:46.071 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-08-24T00:38:46.073 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-08-24T00:38:46.112 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:46 +0000 (0:00:00.071) 0:00:23.177 ********* 2020-08-24T00:38:46.165 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:46.176 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:46.180 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:46.194 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:46.229 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:46 +0000 (0:00:00.116) 0:00:23.294 ********* 2020-08-24T00:38:46.281 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-08-24T00:38:46.283 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:46.323 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:46 +0000 (0:00:00.093) 0:00:23.388 ********* 2020-08-24T00:38:47.235 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:38:51.354 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:47 +0000 (0:00:00.954) 0:00:24.342 ********* TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} Monday 24 August 2020 00:38:47 +0000 (0:00:00.417) 0:00:24.760 ********* TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} Monday 24 August 2020 00:38:48 +0000 (0:00:00.395) 0:00:25.155 ********* TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi060.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} ok: [smithi060.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} Monday 24 August 2020 00:38:48 +0000 (0:00:00.339) 0:00:25.495 ********* TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:48 +0000 (0:00:00.071) 0:00:25.566 ********* TASK [common : Check firewalld status] ***************************************** skipping: [smithi060.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for thi 2020-08-24T00:38:51.356 INFO:teuthology.task.ansible.out:s result", "changed": false} Monday 24 August 2020 00:38:48 +0000 (0:00:00.071) 0:00:25.638 ********* TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:48 +0000 (0:00:00.091) 0:00:25.729 ********* TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} Monday 24 August 2020 00:38:49 +0000 (0:00:00.424) 0:00:26.154 ********* TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-08-24 00:37:32 UTC", "ActiveEnterTimestampMonotonic": "17504030", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target systemd-journald.socket var-run.mount local-fs.target systemd-tmpfiles-setup.service -.mount time-sync.target remote-fs.target basic.target system.slice nss-lookup.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-08-24 00:37:32 UTC", "AssertTimestampMonotonic": "17503028", "Before": "getty@tty1.service xdm.service multi-user.target plymouth-quit.service shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMo 2020-08-24T00:38:51.358 INFO:teuthology.task.ansible.out:de": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-08-24 00:37:32 UTC", "ConditionTimestampMonotonic": "17503028", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-server.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1246", "ExecMainStartTimestamp": "Mon 2020-08-24 00:37:32 UTC", "ExecMainStartTimestampMonotonic": "17503986", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescripto 2020-08-24T00:38:51.360 INFO:teuthology.task.ansible.out:rStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-08-24 00:37:32 UTC", "InactiveExitTimestampMonotonic": "17504030", "InvocationID": "f1c4278aee524fed864990fcee5c0f2f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1246", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": " 2020-08-24T00:38:51.363 INFO:teuthology.task.ansible.out:0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-08-24 00:37:32 UTC", "StateChangeTimestampMonotonic": "17504030", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes 2020-08-24T00:38:51.365 INFO:teuthology.task.ansible.out:", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-08-24 00:37:32 UTC", "WatchdogTimestampMonotonic": "17504029", "WatchdogUSec": "0"}} Monday 24 August 2020 00:38:49 +0000 (0:00:00.790) 0:00:26.944 ********* TASK [common : Get SELinux status] ********************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:49 +0000 (0:00:00.068) 0:00:27.013 ********* TASK [common : nrpe - Install SELinux dependencies] **************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.082) 0:00:27.095 ********* TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.078) 0:00:27.174 ********* TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.075) 0:00:27.249 ********* TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.067) 0:00:27.317 ********* TASK [common : nrpe - Copy SELinux type enforcement file] ******************** 2020-08-24T00:38:51.367 INFO:teuthology.task.ansible.out:** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.068) 0:00:27.386 ********* TASK [common : nrpe - Compile SELinux module file] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.078) 0:00:27.464 ********* TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.073) 0:00:27.538 ********* TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.079) 0:00:27.617 ********* TASK [common : nrpe - Remove temporary files] ********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:38:50 +0000 (0:00:00.074) 0:00:27.692 ********* TASK [testnode : Include package type specific vars.] ************************** ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm 2020-08-24T00:38:51.368 INFO:teuthology.task.ansible.out:-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} Monday 24 August 2020 00:38:50 +0000 (0:00:00.091) 0:00:27.784 ********* TASK [testnode : Including distro specific variables.] ************************* ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipc 2020-08-24T00:38:51.369 INFO:teuthology.task.ansible.out:alc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2020-08-24T00:38:51.424 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:51 +0000 (0:00:00.705) 0:00:28.489 ********* 2020-08-24T00:38:51.485 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2020-08-24T00:38:51.537 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:51 +0000 (0:00:00.112) 0:00:28.601 ********* 2020-08-24T00:38:51.592 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi060.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-08-24T00:38:51.642 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:51 +0000 (0:00:00.106) 0:00:28.707 ********* 2020-08-24T00:38:51.802 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2020-08-24T00:38:51.841 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:51 +0000 (0:00:00.198) 0:00:28.905 ********* 2020-08-24T00:38:52.005 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-08-24T00:38:52.007 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1229, "name": "ubuntu", "state": "present", "system": false} 2020-08-24T00:38:52.046 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:52 +0000 (0:00:00.205) 0:00:29.111 ********* 2020-08-24T00:38:52.082 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-08-24T00:38:52.084 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:52.125 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:52 +0000 (0:00:00.078) 0:00:29.189 ********* 2020-08-24T00:38:52.162 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:52.200 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:52 +0000 (0:00:00.074) 0:00:29.264 ********* 2020-08-24T00:38:52.364 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-08-24T00:38:52.366 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2020-08-24T00:38:52.374 INFO:teuthology.task.ansible.out: 2020-08-24T00:38:52.415 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:52 +0000 (0:00:00.214) 0:00:29.479 ********* 2020-08-24T00:38:52.854 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2020-08-24T00:38:52.901 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:52 +0000 (0:00:00.486) 0:00:29.966 ********* 2020-08-24T00:38:52.902 INFO:teuthology.task.ansible.out: 2020-08-24T00:38:52.945 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-08-24T00:38:52.947 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi060"}, "changed": false} 2020-08-24T00:38:52.990 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:52 +0000 (0:00:00.089) 0:00:30.055 ********* 2020-08-24T00:38:53.954 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-08-24T00:38:53.956 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi060", "ansible_hostname": "smithi060", "ansible_nodename": "smithi060"}, "changed": false, "name": "smithi060"} 2020-08-24T00:38:53.995 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:53 +0000 (0:00:01.004) 0:00:31.060 ********* 2020-08-24T00:38:54.368 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2020-08-24T00:38:54.405 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:54 +0000 (0:00:00.410) 0:00:31.470 ********* 2020-08-24T00:38:54.784 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2020-08-24T00:38:54.829 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:54 +0000 (0:00:00.423) 0:00:31.894 ********* 2020-08-24T00:38:55.510 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** ok: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-08-24T00:38:55.550 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:55 +0000 (0:00:00.720) 0:00:32.615 ********* 2020-08-24T00:38:55.551 INFO:teuthology.task.ansible.out: 2020-08-24T00:38:55.595 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-08-24T00:38:55.596 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:55.639 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:55 +0000 (0:00:00.086) 0:00:32.701 ********* 2020-08-24T00:38:55.669 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-08-24T00:38:55.671 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:55.710 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:55 +0000 (0:00:00.073) 0:00:32.775 ********* 2020-08-24T00:38:55.742 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-08-24T00:38:55.744 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:55.782 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:55 +0000 (0:00:00.071) 0:00:32.847 ********* 2020-08-24T00:38:55.815 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:55.857 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:55 +0000 (0:00:00.075) 0:00:32.922 ********* 2020-08-24T00:38:55.891 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-08-24T00:38:55.893 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:55.934 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:55 +0000 (0:00:00.071) 0:00:32.993 ********* 2020-08-24T00:38:55.964 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-08-24T00:38:55.966 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.004 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:55 +0000 (0:00:00.075) 0:00:33.068 ********* 2020-08-24T00:38:56.046 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.100 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.095) 0:00:33.164 ********* 2020-08-24T00:38:56.143 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-08-24T00:38:56.145 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.180 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.080) 0:00:33.244 ********* 2020-08-24T00:38:56.210 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2020-08-24T00:38:56.250 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.068) 0:00:33.313 ********* 2020-08-24T00:38:56.278 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-08-24T00:38:56.326 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.077) 0:00:33.390 ********* 2020-08-24T00:38:56.357 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-08-24T00:38:56.401 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.075) 0:00:33.466 ********* 2020-08-24T00:38:56.432 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-08-24T00:38:56.434 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.482 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.079) 0:00:33.546 ********* 2020-08-24T00:38:56.530 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi060.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.583 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.102) 0:00:33.648 ********* 2020-08-24T00:38:56.615 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2020-08-24T00:38:56.617 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.654 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.070) 0:00:33.719 ********* 2020-08-24T00:38:56.685 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-08-24T00:38:56.687 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.730 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.076) 0:00:33.795 ********* 2020-08-24T00:38:56.761 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-08-24T00:38:56.763 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.800 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.069) 0:00:33.864 ********* 2020-08-24T00:38:56.836 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-08-24T00:38:56.837 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.874 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.074) 0:00:33.939 ********* 2020-08-24T00:38:56.905 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-08-24T00:38:56.908 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:56.952 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:56 +0000 (0:00:00.077) 0:00:34.017 ********* 2020-08-24T00:38:56.989 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-08-24T00:38:56.991 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.034 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.080) 0:00:34.097 ********* 2020-08-24T00:38:57.063 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-08-24T00:38:57.065 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.102 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.069) 0:00:34.167 ********* 2020-08-24T00:38:57.134 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.174 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.070) 0:00:34.238 ********* 2020-08-24T00:38:57.203 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-08-24T00:38:57.207 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.244 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.070) 0:00:34.308 ********* 2020-08-24T00:38:57.296 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-08-24T00:38:57.298 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.333 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.089) 0:00:34.398 ********* 2020-08-24T00:38:57.375 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-08-24T00:38:57.376 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.414 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.080) 0:00:34.478 ********* 2020-08-24T00:38:57.453 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.488 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.075) 0:00:34.553 ********* 2020-08-24T00:38:57.520 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-08-24T00:38:57.524 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.578 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.086) 0:00:34.639 ********* 2020-08-24T00:38:57.625 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.671 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.089) 0:00:34.729 ********* 2020-08-24T00:38:57.696 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-08-24T00:38:57.700 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.735 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.070) 0:00:34.800 ********* 2020-08-24T00:38:57.766 INFO:teuthology.task.ansible.out: 2020-08-24T00:38:57.768 INFO:teuthology.task.ansible.out:TASK [testnode : Install abrt] ************************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.808 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.073) 0:00:34.873 ********* 2020-08-24T00:38:57.850 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2020-08-24T00:38:57.851 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.901 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.092) 0:00:34.965 ********* 2020-08-24T00:38:57.933 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2020-08-24T00:38:57.935 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:57.976 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:57 +0000 (0:00:00.075) 0:00:35.041 ********* 2020-08-24T00:38:58.012 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:58.054 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:58 +0000 (0:00:00.077) 0:00:35.118 ********* 2020-08-24T00:38:58.085 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2020-08-24T00:38:58.087 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:58.135 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:58 +0000 (0:00:00.080) 0:00:35.199 ********* 2020-08-24T00:38:58.283 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-08-24T00:38:58.285 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005294", "end": "2020-08-24 00:38:58.266142", "rc": 0, "start": "2020-08-24 00:38:58.260848", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:38:58.324 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:58 +0000 (0:00:00.189) 0:00:35.389 ********* 2020-08-24T00:38:58.358 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-08-24T00:38:58.396 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:58 +0000 (0:00:00.071) 0:00:35.460 ********* 2020-08-24T00:38:58.770 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2020-08-24T00:38:58.825 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:58 +0000 (0:00:00.428) 0:00:35.889 ********* 2020-08-24T00:38:58.866 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:38:58.903 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:38:58 +0000 (0:00:00.079) 0:00:35.968 ********* 2020-08-24T00:39:00.280 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* ok: [smithi060.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2020-08-24T00:39:01.318 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2020-08-24T00:39:01.367 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:01 +0000 (0:00:02.463) 0:00:38.432 ********* 2020-08-24T00:39:02.303 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-08-24T00:39:02.305 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:39:02.345 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:02 +0000 (0:00:00.977) 0:00:39.409 ********* 2020-08-24T00:39:02.376 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-08-24T00:39:02.419 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:02 +0000 (0:00:00.074) 0:00:39.484 ********* 2020-08-24T00:39:02.618 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-08-24T00:39:02.620 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.034817", "end": "2020-08-24 00:39:02.592546", "rc": 0, "start": "2020-08-24 00:39:02.557729", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:39:02.664 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:02 +0000 (0:00:00.244) 0:00:39.728 ********* 2020-08-24T00:39:05.496 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:39:05.533 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:05 +0000 (0:00:02.869) 0:00:42.598 ********* 2020-08-24T00:39:15.914 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-08-24T00:39:15.917 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102054 files and directories 2020-08-24T00:39:15.918 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.31) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.31) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.31) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.31) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 0 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2020-08-24T00:39:15.920 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102054 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.31) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.31) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.31) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.31) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} 2020-08-24T00:39:15.960 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:15 +0000 (0:00:10.427) 0:00:53.025 ********* 2020-08-24T00:39:16.893 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:39:16.932 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:16 +0000 (0:00:00.971) 0:00:53.996 ********* 2020-08-24T00:39:16.979 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2020-08-24T00:39:16.981 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:39:17.031 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:39:17 +0000 (0:00:00.099) 0:00:54.096 ********* 2020-08-24T00:39:25.716 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-08-24T00:40:03.745 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.31 [40.4 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.31 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.31 [5190 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.31 [872 kB]\nFetched 10.4 MB in 1s (9877 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2020-08-24T00:40:03.755 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101912 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.31) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.31) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.31) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.31) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.31) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.31) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.31) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.31) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2020-08-24T00:40:03.757 INFO:teuthology.task.ansible.out:1.2) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.2 [2723 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.2 [923 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.31 [40.4 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.31 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.31 [5190 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.31 [872 kB]", "Fetched 10.4 MB in 1s (9877 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2020-08-24T00:40:03.759 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101912 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.2_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.31) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.31) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.31) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.31_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.31) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.2) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.31) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.31) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.31) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.31) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.2) ..."]} Monday 24 August 2020 00:39:25 +0000 (0:00:08.727) 0:01:02.824 ********* TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi060.front.sepia.ceph.com] => 2020-08-24T00:40:03.760 INFO:teuthology.task.ansible.out: {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} Monday 24 August 2020 00:39:26 +0000 (0:00:00.975) 0:01:03.799 ********* TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} Monday 24 August 2020 00:39:26 +0000 (0:00:00.200) 0:01:04.000 ********* TASK [testnode : Enable xattr for this boot.] ********************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:39:27 +0000 (0:00:00.088) 0:01:04.088 ********* TASK [testnode : Ensure fuse group exists.] ************************************ ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} Monday 24 August 2020 00:39:27 +0000 (0:00:00.206) 0:01:04.295 ********* TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} Monday 24 August 2020 00:39:27 +0000 (0:00:00.410) 0:01:04.705 ********* TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ ok: [smithi060.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1229, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "pr 2020-08-24T00:40:03.762 INFO:teuthology.task.ansible.out:esent", "uid": 1000} Monday 24 August 2020 00:39:27 +0000 (0:00:00.221) 0:01:04.927 ********* TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Monday 24 August 2020 00:39:28 +0000 (0:00:00.192) 0:01:05.120 ********* TASK [testnode : Stop apache2] ************************************************* changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Mon 2020-08-24 00:37:32 UTC", "ActiveEnterTimestampMonotonic": "17954057", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target systemd-journald.socket basic.target system.slice systemd-tmpfiles-setup.service sysinit.target -.mount nss-lookup.target remote-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-08-24 00:37:32 UTC", "AssertTimestampMonotonic": "17505128", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_sysl 2020-08-24T00:40:03.764 INFO:teuthology.task.ansible.out:og cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-08-24 00:37:32 UTC", "ConditionTimestampMonotonic": "17505127", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1341", "ExecMainStartTimestamp": "Mon 2020-08-24 00:37:32 UTC", "ExecMainStartTimestampMonotonic": "17954037", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-08-24 00:37:32 UTC", "InactiveExitTimestampMonotonic": "17506541", "InvocationID": "a3bcd41f9 2020-08-24T00:40:03.765 INFO:teuthology.task.ansible.out:b474729a40026661922818a", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1341", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no" 2020-08-24T00:40:03.767 INFO:teuthology.task.ansible.out:, "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-08-24 00:37:32 UTC", "StateChangeTimestampMonotonic": "17954057", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-08-24 00:37:32 UTC", "WatchdogTimestampMonotonic": "17954056", "WatchdogUSec": "0"}} 2020-08-24T00:40:03.812 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:03 +0000 (0:00:35.757) 0:01:40.877 ********* 2020-08-24T00:40:03.872 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-08-24T00:40:03.874 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:03.925 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:03 +0000 (0:00:00.112) 0:01:40.989 ********* 2020-08-24T00:40:03.971 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-08-24T00:40:03.973 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.022 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.097) 0:01:41.087 ********* 2020-08-24T00:40:04.054 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-08-24T00:40:04.056 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.099 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.076) 0:01:41.163 ********* 2020-08-24T00:40:04.149 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-08-24T00:40:04.150 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.197 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.098) 0:01:41.262 ********* 2020-08-24T00:40:04.235 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-08-24T00:40:04.237 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.281 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.083) 0:01:41.345 ********* 2020-08-24T00:40:04.323 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-08-24T00:40:04.325 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.395 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.114) 0:01:41.460 ********* 2020-08-24T00:40:04.437 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-08-24T00:40:04.439 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.476 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.081) 0:01:41.541 ********* 2020-08-24T00:40:04.507 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-08-24T00:40:04.509 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.549 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.071) 0:01:41.614 ********* 2020-08-24T00:40:04.614 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-08-24T00:40:04.616 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.670 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.121) 0:01:41.735 ********* 2020-08-24T00:40:04.707 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-08-24T00:40:04.708 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.744 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.073) 0:01:41.809 ********* 2020-08-24T00:40:04.789 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-08-24T00:40:04.791 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.829 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.084) 0:01:41.894 ********* 2020-08-24T00:40:04.862 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-08-24T00:40:04.864 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.902 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.072) 0:01:41.967 ********* 2020-08-24T00:40:04.933 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-08-24T00:40:04.934 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:04.973 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:04 +0000 (0:00:00.070) 0:01:42.038 ********* 2020-08-24T00:40:05.017 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-08-24T00:40:05.019 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.057 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.084) 0:01:42.122 ********* 2020-08-24T00:40:05.119 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.164 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.107) 0:01:42.229 ********* 2020-08-24T00:40:05.210 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-08-24T00:40:05.218 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.257 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.092) 0:01:42.322 ********* 2020-08-24T00:40:05.291 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-08-24T00:40:05.293 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.329 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.072) 0:01:42.394 ********* 2020-08-24T00:40:05.373 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-08-24T00:40:05.375 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.415 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.085) 0:01:42.479 ********* 2020-08-24T00:40:05.448 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-08-24T00:40:05.466 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.491 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.076) 0:01:42.556 ********* 2020-08-24T00:40:05.524 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-08-24T00:40:05.526 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.572 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.080) 0:01:42.636 ********* 2020-08-24T00:40:05.616 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-08-24T00:40:05.618 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.664 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.092) 0:01:42.729 ********* 2020-08-24T00:40:05.696 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-08-24T00:40:05.698 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.736 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.071) 0:01:42.801 ********* 2020-08-24T00:40:05.767 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-08-24T00:40:05.769 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.812 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.075) 0:01:42.876 ********* 2020-08-24T00:40:05.843 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-08-24T00:40:05.845 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.880 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.068) 0:01:42.945 ********* 2020-08-24T00:40:05.910 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-08-24T00:40:05.912 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:05.948 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:05 +0000 (0:00:00.067) 0:01:43.012 ********* 2020-08-24T00:40:05.988 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-08-24T00:40:05.989 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:06.030 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:06 +0000 (0:00:00.083) 0:01:43.095 ********* 2020-08-24T00:40:06.182 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2020-08-24T00:40:06.218 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:06 +0000 (0:00:00.187) 0:01:43.283 ********* 2020-08-24T00:40:06.581 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2020-08-24T00:40:06.628 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:06 +0000 (0:00:00.409) 0:01:43.693 ********* 2020-08-24T00:40:06.662 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-08-24T00:40:06.664 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:06.699 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:06 +0000 (0:00:00.071) 0:01:43.764 ********* 2020-08-24T00:40:07.085 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-08-24T00:40:07.087 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2020-08-24T00:40:07.146 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:07 +0000 (0:00:00.445) 0:01:44.210 ********* 2020-08-24T00:40:07.526 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-08-24T00:40:07.529 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2020-08-24T00:40:07.564 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:07 +0000 (0:00:00.418) 0:01:44.629 ********* 2020-08-24T00:40:07.929 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-08-24T00:40:07.931 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2020-08-24T00:40:07.969 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:07 +0000 (0:00:00.404) 0:01:45.034 ********* 2020-08-24T00:40:08.120 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2020-08-24T00:40:08.122 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-08-24T00:40:08.159 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.190) 0:01:45.224 ********* 2020-08-24T00:40:08.191 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-08-24T00:40:08.193 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.231 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.071) 0:01:45.296 ********* 2020-08-24T00:40:08.278 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.316 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.085) 0:01:45.381 ********* 2020-08-24T00:40:08.374 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.414 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.097) 0:01:45.479 ********* 2020-08-24T00:40:08.446 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2020-08-24T00:40:08.448 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.495 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.080) 0:01:45.560 ********* 2020-08-24T00:40:08.528 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-08-24T00:40:08.530 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.570 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.075) 0:01:45.635 ********* 2020-08-24T00:40:08.621 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-08-24T00:40:08.655 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.085) 0:01:45.720 ********* 2020-08-24T00:40:08.717 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-08-24T00:40:08.719 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.735 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.737 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.747 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.749 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.770 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-08-24T00:40:08.773 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.781 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.795 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.806 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.815 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.826 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.830 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-08-24T00:40:08.875 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:08 +0000 (0:00:00.219) 0:01:45.940 ********* 2020-08-24T00:40:09.126 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-08-24T00:40:09.128 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.004548", "end": "2020-08-24 00:40:09.090993", "rc": 0, "start": "2020-08-24 00:40:09.086445", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-08-24T00:40:09.173 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:09 +0000 (0:00:00.297) 0:01:46.237 ********* 2020-08-24T00:40:09.213 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-08-24T00:40:09.215 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:09.264 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:09 +0000 (0:00:00.091) 0:01:46.329 ********* 2020-08-24T00:40:09.315 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-08-24T00:40:09.317 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:09.356 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:09 +0000 (0:00:00.089) 0:01:46.418 ********* 2020-08-24T00:40:09.389 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-08-24T00:40:09.393 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:09.426 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:09 +0000 (0:00:00.071) 0:01:46.490 ********* 2020-08-24T00:40:10.356 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-08-24T00:40:10.358 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:40:10.421 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:10 +0000 (0:00:00.993) 0:01:47.484 ********* 2020-08-24T00:40:10.450 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:10.494 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:10 +0000 (0:00:00.069) 0:01:47.553 ********* 2020-08-24T00:40:10.551 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-08-24T00:40:10.552 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13636809, 'uuid': 'f7629cc2-b5ac-46cc-8ac5-d700d34e5986', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226458299, 'size_available': 927573192704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113878, 'block_size': 4096, 'inode_available': 60941098}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226458299, "block_size": 4096, "block_total": 240095108, "block_used": 13636809, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941098, "inode_total": 61054976, "inode_used": 113878, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927573192704, "size_total": 983429562368, "uuid": "f7629cc2-b5ac-46cc-8ac5-d700d34e5986"}} 2020-08-24T00:40:10.594 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:10 +0000 (0:00:00.105) 0:01:47.659 ********* 2020-08-24T00:40:10.745 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-08-24T00:40:10.747 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010950", "end": "2020-08-24 00:40:10.744875", "rc": 0, "start": "2020-08-24 00:40:10.733925", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:10.791 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:10 +0000 (0:00:00.197) 0:01:47.856 ********* 2020-08-24T00:40:10.838 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13636809, 'uuid': 'f7629cc2-b5ac-46cc-8ac5-d700d34e5986', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226458299, 'size_available': 927573192704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113878, 'block_size': 4096, 'inode_available': 60941098}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226458299, "block_size": 4096, "block_total": 240095108, "block_used": 13636809, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941098, "inode_total": 61054976, "inode_used": 113878, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927573192704, "size_total": 983429562368, "uuid": "f7629cc2-b5ac-46cc-8ac5-d700d34e5986"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:10.880 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:10 +0000 (0:00:00.088) 0:01:47.944 ********* 2020-08-24T00:40:10.910 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-08-24T00:40:10.953 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:10 +0000 (0:00:00.072) 0:01:48.017 ********* 2020-08-24T00:40:11.004 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-08-24T00:40:11.006 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:12.358 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAu3h1ayTrv6zXw4JQl7buxPZhTvXWEiOZ'], 'uuids': ['636209e0-46de-4581-bfab-77cd3a4b4ab2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.217734", "end": "2020-08-24 00:40:12.352513", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAu3h1ayTrv6zXw4JQl7buxPZhTvXWEiOZ"], "labels": [], "masters": [], "uuids": ["636209e0-46de-4581-bfab-77cd3a4b4ab2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:11.134779", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-08-24T00:40:12.370 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:12.381 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:13.669 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAfqt7l7jNyS0u8i2I0PVPUQ0IuNqJpFdf'], 'uuids': ['990d0b73-5c0e-49bc-a209-abd3e09249d6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.152279", "end": "2020-08-24 00:40:13.661766", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAfqt7l7jNyS0u8i2I0PVPUQ0IuNqJpFdf"], "labels": [], "masters": [], "uuids": ["990d0b73-5c0e-49bc-a209-abd3e09249d6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:12.509487", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-08-24T00:40:13.682 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED', 'wwn-0x5000c500871b1a82'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'f7629cc2-b5ac-46cc-8ac5-d700d34e5986', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1', 'wwn-0x5000c500871b1a82-part1'], 'uuids': ['f7629cc2-b5ac-46cc-8ac5-d700d34e5986']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b1a82', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H4ED', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED", "wwn-0x5000c500871b1a82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1", "wwn-0x5000c500871b1a82-part1"], "labels": [], "masters": [], "uuids": ["f7629cc2-b5ac-46cc-8ac5-d700d34e5986"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "f7629cc2-b5ac-46cc-8ac5-d700d34e5986"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H4ED", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b1a82"}}, "skip_reason": 2020-08-24T00:40:13.684 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-08-24T00:40:14.973 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-29aumt-yHig-MfTU-ltZg-bytu-4WMh-OCFfwo', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN', 'nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.150457", "end": "2020-08-24 00:40:14.960398", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-29aumt-yHig-MfTU-ltZg-bytu-4WMh-OCFfwo", "nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN", "nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:13.809941", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2020-08-24T00:40:14.975 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-08-24T00:40:14.991 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:14.996 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:16.250 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAHhvD3LTxdq4N7zjfhyrJRhSXp8al9jp5'], 'uuids': ['7b01a240-9029-410b-a97c-f23f3deb3585']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.129245", "end": "2020-08-24 00:40:16.245193", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAHhvD3LTxdq4N7zjfhyrJRhSXp8al9jp5"], "labels": [], "masters": [], "uuids": ["7b01a240-9029-410b-a97c-f23f3deb3585"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:15.115948", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-08-24T00:40:16.262 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:16.284 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:17.541 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FArCCsd83KtObHKWdgAr4eUUV1sL5VKO3Y'], 'uuids': ['90b7cd51-d4c8-47fc-b693-c7e222f8e4b4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.124225", "end": "2020-08-24 00:40:17.528736", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FArCCsd83KtObHKWdgAr4eUUV1sL5VKO3Y"], "labels": [], "masters": [], "uuids": ["90b7cd51-d4c8-47fc-b693-c7e222f8e4b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:16.404511", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-08-24T00:40:17.542 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:18.805 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAjTuJ3Hqf7hTHBfhf71eiDzfLgUbSLydf'], 'uuids': ['8fcfb840-790a-4b9e-a0cd-f47e9e081559']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.120671", "end": "2020-08-24 00:40:18.795866", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAjTuJ3Hqf7hTHBfhf71eiDzfLgUbSLydf"], "labels": [], "masters": [], "uuids": ["8fcfb840-790a-4b9e-a0cd-f47e9e081559"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:17.675195", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-08-24T00:40:18.856 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:18 +0000 (0:00:07.902) 0:01:55.920 ********* 2020-08-24T00:40:18.924 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:19.169 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAu3h1ayTrv6zXw4JQl7buxPZhTvXWEiOZ'], 'uuids': ['636209e0-46de-4581-bfab-77cd3a4b4ab2']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.113812", "end": "2020-08-24 00:40:19.161940", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAu3h1ayTrv6zXw4JQl7buxPZhTvXWEiOZ"], "labels": [], "masters": [], "uuids": ["636209e0-46de-4581-bfab-77cd3a4b4ab2"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:19.048128", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0990829 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0990829 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:19.181 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:19.192 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:19.431 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAfqt7l7jNyS0u8i2I0PVPUQ0IuNqJpFdf'], 'uuids': ['990d0b73-5c0e-49bc-a209-abd3e09249d6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.107248", "end": "2020-08-24 00:40:19.422075", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAfqt7l7jNyS0u8i2I0PVPUQ0IuNqJpFdf"], "labels": [], "masters": [], "uuids": ["990d0b73-5c0e-49bc-a209-abd3e09249d6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:19.314827", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.102662 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.102662 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:19.443 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED', 'wwn-0x5000c500871b1a82'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'f7629cc2-b5ac-46cc-8ac5-d700d34e5986', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1', 'wwn-0x5000c500871b1a82-part1'], 'uuids': ['f7629cc2-b5ac-46cc-8ac5-d700d34e5986']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b1a82', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4H4ED', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED", "wwn-0x5000c500871b1a82"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4H4ED-part1", "wwn-0x5000c500871b1a82-part1"], "labels": [], "masters": [], "uuids": ["f7629cc2-b5ac-46cc-8ac5-d700d34e5986"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "f7629cc2-b5ac-46cc-8ac5-d700d34e5986"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4H4ED", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b1a82"}}, "skip_reason": 2020-08-24T00:40:19.445 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2020-08-24T00:40:19.615 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-29aumt-yHig-MfTU-ltZg-bytu-4WMh-OCFfwo', 'nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN', 'nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.047747", "end": "2020-08-24 00:40:19.604852", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-29aumt-yHig-MfTU-ltZg-bytu-4WMh-OCFfwo", "nvme-INTEL_SSDPED1D480GA_PHMB7476008A480DGN", "nvme-nvme.8086-50484d42373437363030384134383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:19.557105", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0448791 s, 2.6 GB/s", "stderr_lines": [" 2020-08-24T00:40:19.617 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0448791 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:19.627 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:19.638 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:19.859 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAHhvD3LTxdq4N7zjfhyrJRhSXp8al9jp5'], 'uuids': ['7b01a240-9029-410b-a97c-f23f3deb3585']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.106791", "end": "2020-08-24 00:40:19.850143", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAHhvD3LTxdq4N7zjfhyrJRhSXp8al9jp5"], "labels": [], "masters": [], "uuids": ["7b01a240-9029-410b-a97c-f23f3deb3585"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:19.743352", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.103782 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.103782 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:19.870 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:19.881 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:20.123 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FArCCsd83KtObHKWdgAr4eUUV1sL5VKO3Y'], 'uuids': ['90b7cd51-d4c8-47fc-b693-c7e222f8e4b4']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.110237", "end": "2020-08-24 00:40:20.108719", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FArCCsd83KtObHKWdgAr4eUUV1sL5VKO3Y"], "labels": [], "masters": [], "uuids": ["90b7cd51-d4c8-47fc-b693-c7e222f8e4b4"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:19.998482", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.105535 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.105535 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:20.134 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:20.363 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAjTuJ3Hqf7hTHBfhf71eiDzfLgUbSLydf'], 'uuids': ['8fcfb840-790a-4b9e-a0cd-f47e9e081559']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.111392", "end": "2020-08-24 00:40:20.363925", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-RCmP2o88XNuh3o8W8xdTI8yOLqnPb8FAjTuJ3Hqf7hTHBfhf71eiDzfLgUbSLydf"], "labels": [], "masters": [], "uuids": ["8fcfb840-790a-4b9e-a0cd-f47e9e081559"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-08-24 00:40:20.252533", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.10678 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.10678 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:20.422 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:20 +0000 (0:00:01.566) 0:01:57.487 ********* 2020-08-24T00:40:20.727 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-08-24T00:40:20.729 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.142597", "end": "2020-08-24 00:40:20.712467", "rc": 0, "start": "2020-08-24 00:40:20.569870", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:20.772 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:20 +0000 (0:00:00.349) 0:01:57.836 ********* 2020-08-24T00:40:20.970 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-08-24T00:40:20.972 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.045918", "end": "2020-08-24 00:40:20.966352", "rc": 0, "start": "2020-08-24 00:40:20.920434", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:21.018 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.246) 0:01:58.083 ********* 2020-08-24T00:40:21.049 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-08-24T00:40:21.095 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.077) 0:01:58.160 ********* 2020-08-24T00:40:21.135 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-08-24T00:40:21.137 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.187 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.091) 0:01:58.252 ********* 2020-08-24T00:40:21.218 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-08-24T00:40:21.226 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.261 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.073) 0:01:58.325 ********* 2020-08-24T00:40:21.301 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-08-24T00:40:21.303 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.349 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.088) 0:01:58.414 ********* 2020-08-24T00:40:21.382 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-08-24T00:40:21.384 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.439 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.089) 0:01:58.503 ********* 2020-08-24T00:40:21.586 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-08-24T00:40:21.588 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-08-24T00:40:21.625 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.186) 0:01:58.690 ********* 2020-08-24T00:40:21.674 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-08-24T00:40:21.676 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'block_used': 13636809, 'uuid': 'f7629cc2-b5ac-46cc-8ac5-d700d34e5986', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226458299, 'size_available': 927573192704, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113878, 'block_size': 4096, 'inode_available': 60941098}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226458299, "block_size": 4096, "block_total": 240095108, "block_used": 13636809, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941098, "inode_total": 61054976, "inode_used": 113878, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927573192704, "size_total": 983429562368, "uuid": "f7629cc2-b5ac-46cc-8ac5-d700d34e5986"}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.715 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.090) 0:01:58.780 ********* 2020-08-24T00:40:21.758 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-08-24T00:40:21.760 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.809 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.093) 0:01:58.874 ********* 2020-08-24T00:40:21.841 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-08-24T00:40:21.843 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.883 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.073) 0:01:58.947 ********* 2020-08-24T00:40:21.914 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-08-24T00:40:21.916 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:21.952 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:21 +0000 (0:00:00.070) 0:01:59.017 ********* 2020-08-24T00:40:22.002 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:22.036 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:22 +0000 (0:00:00.084) 0:01:59.101 ********* 2020-08-24T00:40:22.080 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-08-24T00:40:22.082 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:22.125 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:22 +0000 (0:00:00.088) 0:01:59.190 ********* 2020-08-24T00:40:22.167 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-08-24T00:40:22.169 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:22.210 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:22 +0000 (0:00:00.084) 0:01:59.274 ********* 2020-08-24T00:40:22.879 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-08-24T00:40:22.881 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-08-24T00:40:22.933 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:22 +0000 (0:00:00.722) 0:01:59.997 ********* 2020-08-24T00:40:23.544 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-08-24T00:40:23.546 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-08-24T00:40:23.796 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-08-24T00:40:24.054 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-08-24T00:40:24.310 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-08-24T00:40:24.567 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-08-24T00:40:24.614 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:24 +0000 (0:00:01.675) 0:02:01.673 ********* 2020-08-24T00:40:24.771 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-08-24T00:40:24.773 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-08-24T00:40:24.810 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:24 +0000 (0:00:00.201) 0:02:01.874 ********* 2020-08-24T00:40:24.994 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-08-24T00:40:24.997 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-08-24T00:40:25.122 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-08-24T00:40:25.252 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-08-24T00:40:25.385 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-08-24T00:40:25.389 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:25.437 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:25 +0000 (0:00:00.627) 0:02:02.502 ********* 2020-08-24T00:40:25.589 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-08-24T00:40:25.591 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2020-08-24T00:40:25.650 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:25 +0000 (0:00:00.210) 0:02:02.713 ********* 2020-08-24T00:40:25.693 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-08-24T00:40:25.695 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-08-24T00:40:25.737 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:25 +0000 (0:00:00.089) 0:02:02.802 ********* 2020-08-24T00:40:26.227 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi060.front.sepia.ceph.com] => {"changed": true} 2020-08-24T00:40:26.266 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:26 +0000 (0:00:00.529) 0:02:03.331 ********* 2020-08-24T00:40:26.816 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-08-24T00:40:26.866 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:26 +0000 (0:00:00.599) 0:02:03.930 ********* 2020-08-24T00:40:27.042 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-08-24T00:40:27.044 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1229, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2020-08-24T00:40:27.099 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:27 +0000 (0:00:00.233) 0:02:04.163 ********* 2020-08-24T00:40:27.264 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-08-24T00:40:27.266 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2020-08-24T00:40:27.304 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:27 +0000 (0:00:00.205) 0:02:04.369 ********* 2020-08-24T00:40:27.678 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-08-24T00:40:27.680 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1229, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2020-08-24T00:40:27.722 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:27 +0000 (0:00:00.415) 0:02:04.785 ********* 2020-08-24T00:40:28.096 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-08-24T00:40:28.098 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2020-08-24T00:40:28.152 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:28 +0000 (0:00:00.431) 0:02:05.216 ********* 2020-08-24T00:40:29.091 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:40:29.134 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:29 +0000 (0:00:00.978) 0:02:06.194 ********* 2020-08-24T00:40:29.340 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-08-24T00:40:29.342 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047860", "end": "2020-08-24 00:40:29.337203", "rc": 0, "start": "2020-08-24 00:40:29.289343", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2020-08-24T00:40:29.383 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:29 +0000 (0:00:00.253) 0:02:06.448 ********* 2020-08-24T00:40:29.436 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:29.465 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:29 +0000 (0:00:00.081) 0:02:06.529 ********* 2020-08-24T00:40:29.495 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-08-24T00:40:29.497 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:29.534 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:29 +0000 (0:00:00.068) 0:02:06.598 ********* 2020-08-24T00:40:30.445 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-08-24T00:40:30.447 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:40:30.483 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:30 +0000 (0:00:00.949) 0:02:07.547 ********* 2020-08-24T00:40:30.524 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-08-24T00:40:30.526 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:30.561 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:30 +0000 (0:00:00.078) 0:02:07.626 ********* 2020-08-24T00:40:30.952 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-08-24T00:40:30.954 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2020-08-24T00:40:31.011 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:31 +0000 (0:00:00.450) 0:02:08.076 ********* 2020-08-24T00:40:31.043 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-08-24T00:40:31.045 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:31.080 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:31 +0000 (0:00:00.068) 0:02:08.145 ********* 2020-08-24T00:40:31.326 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-08-24T00:40:31.328 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2020-08-24 00:37:38 UTC", "ActiveEnterTimestampMonotonic": "23447622", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount systemd-journald.socket system.slice systemd-tmpfiles-setup.service basic.target network.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2020-08-24 00:37:31 UTC", "AssertTimestampMonotonic": "17024159", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2020-08-24 00:37:31 UTC", "ConditionTimestampMonotonic": "17024159", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2020-08-24T00:40:31.329 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1888", "ExecMainStartTimestamp": "Mon 2020-08-24 00:37:38 UTC", "ExecMainStartTimestampMonotonic": "23447591", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2020-08-24 00:37:31 UTC", "InactiveExitTimestampMonotonic": "17025057", "InvocationID": "6bce5714ee7c411d9d7446cbb100e26e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128055", "LimitNPROCSoft": "128055", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2020-08-24T00:40:31.331 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128055", "LimitSIGPENDINGSoft": "128055", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1888", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2020-08-24 00:37:38 UTC", "S 2020-08-24T00:40:31.333 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23447622", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2020-08-24 00:37:38 UTC", "WatchdogTimestampMonotonic": "23447620", "WatchdogUSec": "0"}} 2020-08-24T00:40:31.374 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:31 +0000 (0:00:00.293) 0:02:08.439 ********* 2020-08-24T00:40:31.415 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-08-24T00:40:31.418 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2020-08-24T00:40:31.469 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:31 +0000 (0:00:00.095) 0:02:08.534 ********* 2020-08-24T00:40:31.500 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-08-24T00:40:31.502 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:31.549 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:31 +0000 (0:00:00.080) 0:02:08.614 ********* 2020-08-24T00:40:31.582 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-08-24T00:40:31.585 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:31.619 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:31 +0000 (0:00:00.069) 0:02:08.684 ********* 2020-08-24T00:40:32.563 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-08-24T00:40:32.566 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"cache_update_time": 1598229485, "cache_updated": false, "changed": false} 2020-08-24T00:40:32.602 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:32 +0000 (0:00:00.982) 0:02:09.666 ********* 2020-08-24T00:40:32.645 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:32.685 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:32 +0000 (0:00:00.083) 0:02:09.750 ********* 2020-08-24T00:40:32.854 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "gid": 1229, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2020-08-24T00:40:32.887 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:32 +0000 (0:00:00.201) 0:02:09.952 ********* 2020-08-24T00:40:33.298 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-08-24T00:40:33.300 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1229, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2020-08-24T00:40:33.338 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:33 +0000 (0:00:00.450) 0:02:10.403 ********* 2020-08-24T00:40:34.447 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-08-24T00:40:34.486 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:34 +0000 (0:00:01.147) 0:02:11.550 ********* 2020-08-24T00:40:35.043 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi060.front.sepia.ceph.com] => {"changed": false} 2020-08-24T00:40:35.094 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:35 +0000 (0:00:00.607) 0:02:12.158 ********* 2020-08-24T00:40:35.136 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2020-08-24T00:40:35.173 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:35 +0000 (0:00:00.079) 0:02:12.238 ********* 2020-08-24T00:40:35.342 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-08-24T00:40:35.344 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1598229467.4798715, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1598229467.4758694, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 3001, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1598229467.4758694, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "168368194", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2020-08-24T00:40:35.386 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:35 +0000 (0:00:00.207) 0:02:12.446 ********* 2020-08-24T00:40:35.870 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-08-24T00:40:35.877 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2020-08-24T00:40:35.921 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:35 +0000 (0:00:00.540) 0:02:12.986 ********* 2020-08-24T00:40:35.975 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-08-24T00:40:35.977 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-08-24T00:40:36.020 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:36 +0000 (0:00:00.098) 0:02:13.085 ********* 2020-08-24T00:40:36.055 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-08-24T00:40:36.057 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:36.094 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:36 +0000 (0:00:00.073) 0:02:13.158 ********* 2020-08-24T00:40:36.245 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-08-24T00:40:36.247 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-08-24T00:40:36.287 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:36 +0000 (0:00:00.192) 0:02:13.351 ********* 2020-08-24T00:40:36.431 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* changed: [smithi060.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-08-24T00:40:37.065 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:36 +0000 (0:00:00.182) 0:02:13.533 ********* TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:40:36 +0000 (0:00:00.075) 0:02:13.609 ********* TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi060.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:40:36 +0000 (0:00:00.102) 0:02:13.712 ********* TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason' 2020-08-24T00:40:37.067 INFO:teuthology.task.ansible.out:: 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi060.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:40:36 +0000 (0:00:00.132) 0:02:13.844 ********* TASK [testnode : set_fact] ***************************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:40:36 +0000 (0:00:00.080) 0:02:13.924 ********* TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ******************** 2020-08-24T00:40:37.070 INFO:teuthology.task.ansible.out:** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:40:36 +0000 (0:00:00.092) 0:02:14.016 ********* TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 24 August 2020 00:40:37 +0000 (0:00:00.085) 0:02:14.102 ********* TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:37.114 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:37 +0000 (0:00:00.076) 0:02:14.179 ********* 2020-08-24T00:40:37.156 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-08-24T00:40:37.158 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:37.202 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:37 +0000 (0:00:00.088) 0:02:14.267 ********* 2020-08-24T00:40:37.233 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-08-24T00:40:37.234 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:37.275 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:37 +0000 (0:00:00.072) 0:02:14.339 ********* 2020-08-24T00:40:37.306 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-08-24T00:40:37.308 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:37.347 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:37 +0000 (0:00:00.072) 0:02:14.412 ********* 2020-08-24T00:40:37.388 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-08-24T00:40:37.390 INFO:teuthology.task.ansible.out:skipping: [smithi060.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-08-24T00:40:37.429 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:37 +0000 (0:00:00.082) 0:02:14.494 ********* 2020-08-24T00:40:37.609 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-08-24T00:40:37.610 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.010980", "end": "2020-08-24 00:40:37.581537", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-08-24 00:40:37.570557", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:37.737 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006168", "end": "2020-08-24 00:40:37.724551", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-08-24 00:40:37.718383", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:37.865 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005621", "end": "2020-08-24 00:40:37.857221", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-08-24 00:40:37.851600", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:38.018 INFO:teuthology.task.ansible.out:ok: [smithi060.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.028339", "end": "2020-08-24 00:40:38.011037", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-08-24 00:40:37.982698", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-08-24T00:40:38.062 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:38 +0000 (0:00:00.632) 0:02:15.127 ********* 2020-08-24T00:40:38.100 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-08-24T00:40:38.137 INFO:teuthology.task.ansible.out:Monday 24 August 2020 00:40:38 +0000 (0:00:00.075) 0:02:15.202 ********* 2020-08-24T00:40:38.275 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-08-24T00:40:38.277 INFO:teuthology.task.ansible.out:changed: [smithi060.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2020-08-24T00:40:38.376 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2020-08-24T00:40:38.381 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-08-24T00:40:38.514 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi060.front.sepia.ceph.com : ok=108 changed=26 unreachable=0 failed=0 skipped=181 rescued=0 ignored=0 Monday 24 August 2020 00:40:38 +0000 (0:00:00.375) 0:02:15.577 ********* =============================================================================== testnode : Stop apache2 ------------------------------------------------ 35.76s testnode : Ensure packages are not present. ---------------------------- 10.43s testnode : Install packages --------------------------------------------- 8.73s testnode : Zap all non-root disks --------------------------------------- 7.90s common : Update apt cache ----------------------------------------------- 3.27s testnode : Update apt cache. -------------------------------------------- 2.87s testnode : Install apt keys --------------------------------------------- 2.46s Gathering Facts --------------------------------------------------------- 1.92s testnode : Create logical volume(s) ------------------------------------- 1.68s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.57s testnode : Install packages via pip ------------------------------------- 1.15s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.11s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.07s testnode : Set the system's hostname ------------------------------------ 1.00s testnode : Make sure apt dependencies are installed --------------------- 0.99s testnode : Install python-pip on apt based systems. --------------------- 0.98s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.98s testnode : Install python-apt ------------------------------------------- 0.98s testnode : Install non aar 2020-08-24T00:40:38.516 INFO:teuthology.task.ansible.out:ch64 packages --------------------------------- 0.98s testnode : Upgrade packages --------------------------------------------- 0.97s 2020-08-24T00:40:38.861 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi060.front.sepia.ceph.com')] 2020-08-24T00:40:38.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-08-24T00:40:38.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi060.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2020-08-24T00:40:38.945 INFO:teuthology.orchestra.run.smithi060:> true 2020-08-24T00:40:39.143 INFO:teuthology.run_tasks:Running task clock... 2020-08-24T00:40:39.156 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-08-24T00:40:39.156 INFO:teuthology.orchestra.run:Running command with timeout 360 2020-08-24T00:40:39.157 INFO:teuthology.orchestra.run.smithi060:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-08-24T00:40:39.232 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: ntpd 4.2.8p10@1.3728-o (1): Starting 2020-08-24T00:40:39.232 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Command line: ntpd -gq 2020-08-24T00:40:39.234 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: proto: precision = 0.126 usec (-23) 2020-08-24T00:40:39.234 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-08-24T00:40:39.234 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: restrict ::: KOD does nothing without LIMITED. 2020-08-24T00:40:39.235 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Listen and drop on 0 v6wildcard [::]:123 2020-08-24T00:40:39.235 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2020-08-24T00:40:39.235 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Listen normally on 2 lo 127.0.0.1:123 2020-08-24T00:40:39.235 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Listen normally on 3 ens1f0 172.21.15.60:123 2020-08-24T00:40:39.235 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Listen normally on 4 lo [::1]:123 2020-08-24T00:40:39.236 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:1560%3]:123 2020-08-24T00:40:39.236 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:39 ntpd[6848]: Listening on routing socket on fd #22 for interface updates 2020-08-24T00:40:39.236 INFO:teuthology.orchestra.run.smithi060.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2020-08-24T00:40:39.236 INFO:teuthology.orchestra.run.smithi060.stderr:restrict ::: KOD does nothing without LIMITED. 2020-08-24T00:40:42.233 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:42 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:42 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:42 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:42.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:42 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:44.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:44.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:44.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:44.237 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:44 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:46.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:46.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:46.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:46.237 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:46.237 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:46.237 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:46 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:48.233 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:48.234 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2020-08-24T00:40:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: can't open /var/log/ntpstats/rawstats.20200824: Permission denied 2020-08-24T00:40:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2020-08-24T00:40:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: can't open /var/log/ntpstats/peerstats.20200824: Permission denied 2020-08-24T00:40:48.235 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2020-08-24T00:40:48.236 INFO:teuthology.orchestra.run.smithi060.stderr:24 Aug 00:40:48 ntpd[6848]: can't open /var/log/ntpstats/loopstats.20200824: Permission denied 2020-08-24T00:40:48.236 INFO:teuthology.orchestra.run.smithi060.stdout:24 Aug 00:40:48 ntpd[6848]: ntpd: time slew -0.021845 s 2020-08-24T00:40:48.236 INFO:teuthology.orchestra.run.smithi060.stdout:ntpd: time slew -0.021845s 2020-08-24T00:40:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: remote refid st t when poll reach delay offset jitter 2020-08-24T00:40:48.288 INFO:teuthology.orchestra.run.smithi060.stdout:============================================================================== 2020-08-24T00:40:48.288 INFO:teuthology.orchestra.run.smithi060.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-08-24T00:40:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-08-24T00:40:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-08-24T00:40:48.289 INFO:teuthology.orchestra.run.smithi060.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2020-08-24T00:40:48.290 INFO:teuthology.run_tasks:Running task install... 2020-08-24T00:40:48.327 DEBUG:teuthology.task.install:project ceph 2020-08-24T00:40:48.328 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '0a941e55043ccee6d35a8717bec55ff7581e67d8'}} 2020-08-24T00:40:48.329 DEBUG:teuthology.task.install:config {'sha1': '0a941e55043ccee6d35a8717bec55ff7581e67d8'} 2020-08-24T00:40:48.329 INFO:teuthology.task.install:Using flavor: basic 2020-08-24T00:40:48.336 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-08-24T00:40:48.336 INFO:teuthology.task.install:extra packages: [] 2020-08-24T00:40:48.337 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-08-24T00:40:48.404 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-08-24T00:40:48.406 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-08-24T00:40:48.406 INFO:teuthology.orchestra.run.smithi060:> sudo apt-key list | grep Ceph 2020-08-24T00:40:48.490 INFO:teuthology.orchestra.run.smithi060.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2020-08-24T00:40:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2020-08-24T00:40:48.547 INFO:teuthology.orchestra.run.smithi060.stdout:uid [ unknown] Ceph.com (release key) 2020-08-24T00:40:48.549 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-08-24T00:40:48.601 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-08-24T00:40:48.611 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-08-24T00:40:48.611 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2020-08-24T00:40:48.612 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=0a941e55043ccee6d35a8717bec55ff7581e67d8 2020-08-24T00:40:48.819 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default/ 2020-08-24T00:40:48.820 INFO:teuthology.task.install.deb:Package version is 15.2.4-661-g0a941e5-1bionic 2020-08-24T00:40:48.957 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /etc/apt/sources.list.d/ceph.list' 2020-08-24T00:40:49.004 INFO:teuthology.orchestra.run.smithi060:> sudo apt-get update 2020-08-24T00:40:49.188 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2020-08-24T00:40:49.284 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:2 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic InRelease 2020-08-24T00:40:49.304 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic InRelease 2020-08-24T00:40:49.307 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2020-08-24T00:40:49.332 INFO:teuthology.orchestra.run.smithi060.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic Release.gpg 2020-08-24T00:40:49.355 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,100 B] 2020-08-24T00:40:49.384 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2020-08-24T00:40:49.389 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2020-08-24T00:40:49.464 INFO:teuthology.orchestra.run.smithi060.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2020-08-24T00:40:49.790 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 26.6 kB in 1s (46.1 kB/s) 2020-08-24T00:40:51.047 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-08-24T00:40:51.095 INFO:teuthology.orchestra.run.smithi060:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.4-661-g0a941e5-1bionic cephadm=15.2.4-661-g0a941e5-1bionic ceph-mds=15.2.4-661-g0a941e5-1bionic ceph-mgr=15.2.4-661-g0a941e5-1bionic ceph-common=15.2.4-661-g0a941e5-1bionic ceph-fuse=15.2.4-661-g0a941e5-1bionic ceph-test=15.2.4-661-g0a941e5-1bionic radosgw=15.2.4-661-g0a941e5-1bionic python3-rados=15.2.4-661-g0a941e5-1bionic python3-rgw=15.2.4-661-g0a941e5-1bionic python3-cephfs=15.2.4-661-g0a941e5-1bionic python3-rbd=15.2.4-661-g0a941e5-1bionic libcephfs2=15.2.4-661-g0a941e5-1bionic libcephfs-dev=15.2.4-661-g0a941e5-1bionic librados2=15.2.4-661-g0a941e5-1bionic librbd1=15.2.4-661-g0a941e5-1bionic rbd-fuse=15.2.4-661-g0a941e5-1bionic 2020-08-24T00:40:51.176 INFO:teuthology.orchestra.run.smithi060.stdout:Reading package lists... 2020-08-24T00:40:51.432 INFO:teuthology.orchestra.run.smithi060.stdout:Building dependency tree... 2020-08-24T00:40:51.433 INFO:teuthology.orchestra.run.smithi060.stdout:Reading state information... 2020-08-24T00:40:51.610 INFO:teuthology.orchestra.run.smithi060.stdout:The following additional packages will be installed: 2020-08-24T00:40:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-08-24T00:40:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-08-24T00:40:51.610 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2020-08-24T00:40:51.611 INFO:teuthology.orchestra.run.smithi060.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2020-08-24T00:40:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: libsnappy1v5 libxslt1.1 python-attr python-funcsigs python-pastedeploy-tpl 2020-08-24T00:40:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: python-pluggy python-py python-pytest python3-asn1crypto python3-attr 2020-08-24T00:40:51.612 INFO:teuthology.orchestra.run.smithi060.stdout: python3-bcrypt python3-bs4 python3-ceph-argparse python3-ceph-common 2020-08-24T00:40:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-08-24T00:40:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-08-24T00:40:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-08-24T00:40:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-08-24T00:40:51.613 INFO:teuthology.orchestra.run.smithi060.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-08-24T00:40:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-08-24T00:40:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-08-24T00:40:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-24T00:40:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-24T00:40:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytest python3-repoze.lru python3-routes python3-rsa python3-scipy 2020-08-24T00:40:51.614 INFO:teuthology.orchestra.run.smithi060.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2020-08-24T00:40:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2020-08-24T00:40:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2020-08-24T00:40:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2020-08-24T00:40:51.615 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2020-08-24T00:40:51.616 INFO:teuthology.orchestra.run.smithi060.stdout:Suggested packages: 2020-08-24T00:40:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2020-08-24T00:40:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: python-pytest-xdist python-mock python-cryptography-doc 2020-08-24T00:40:51.616 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2020-08-24T00:40:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2020-08-24T00:40:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2020-08-24T00:40:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2020-08-24T00:40:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2020-08-24T00:40:51.617 INFO:teuthology.orchestra.run.smithi060.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2020-08-24T00:40:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2020-08-24T00:40:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2020-08-24T00:40:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2020-08-24T00:40:51.618 INFO:teuthology.orchestra.run.smithi060.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2020-08-24T00:40:51.619 INFO:teuthology.orchestra.run.smithi060.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2020-08-24T00:40:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2020-08-24T00:40:51.620 INFO:teuthology.orchestra.run.smithi060.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2020-08-24T00:40:51.717 INFO:teuthology.orchestra.run.smithi060.stdout:The following NEW packages will be installed: 2020-08-24T00:40:51.717 INFO:teuthology.orchestra.run.smithi060.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2020-08-24T00:40:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2020-08-24T00:40:51.718 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2020-08-24T00:40:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2020-08-24T00:40:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2020-08-24T00:40:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 python-attr 2020-08-24T00:40:51.720 INFO:teuthology.orchestra.run.smithi060.stdout: python-funcsigs python-pastedeploy-tpl python-pluggy python-py python-pytest 2020-08-24T00:40:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2020-08-24T00:40:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2020-08-24T00:40:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2020-08-24T00:40:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2020-08-24T00:40:51.721 INFO:teuthology.orchestra.run.smithi060.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2020-08-24T00:40:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2020-08-24T00:40:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2020-08-24T00:40:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2020-08-24T00:40:51.722 INFO:teuthology.orchestra.run.smithi060.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2020-08-24T00:40:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2020-08-24T00:40:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2020-08-24T00:40:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2020-08-24T00:40:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2020-08-24T00:40:51.723 INFO:teuthology.orchestra.run.smithi060.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2020-08-24T00:40:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2020-08-24T00:40:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2020-08-24T00:40:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2020-08-24T00:40:51.724 INFO:teuthology.orchestra.run.smithi060.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2020-08-24T00:40:51.724 INFO:teuthology.orchestra.run.smithi060.stdout:The following packages will be upgraded: 2020-08-24T00:40:51.725 INFO:teuthology.orchestra.run.smithi060.stdout: librados2 librbd1 2020-08-24T00:40:51.878 INFO:teuthology.orchestra.run.smithi060.stdout:2 upgraded, 110 newly installed, 0 to remove and 0 not upgraded. 2020-08-24T00:40:51.878 INFO:teuthology.orchestra.run.smithi060.stdout:Need to get 157 MB of archives. 2020-08-24T00:40:51.878 INFO:teuthology.orchestra.run.smithi060.stdout:After this operation, 730 MB of additional disk space will be used. 2020-08-24T00:40:51.878 INFO:teuthology.orchestra.run.smithi060.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.4-661-g0a941e5-1bionic [1,629 kB] 2020-08-24T00:40:51.936 INFO:teuthology.orchestra.run.smithi060.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2020-08-24T00:40:52.017 INFO:teuthology.orchestra.run.smithi060.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2020-08-24T00:40:52.058 INFO:teuthology.orchestra.run.smithi060.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.4-661-g0a941e5-1bionic [3,051 kB] 2020-08-24T00:40:52.149 INFO:teuthology.orchestra.run.smithi060.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.4-661-g0a941e5-1bionic [433 kB] 2020-08-24T00:40:52.163 INFO:teuthology.orchestra.run.smithi060.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.4-661-g0a941e5-1bionic [331 kB] 2020-08-24T00:40:52.173 INFO:teuthology.orchestra.run.smithi060.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.4-661-g0a941e5-1bionic [20.6 kB] 2020-08-24T00:40:52.174 INFO:teuthology.orchestra.run.smithi060.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.4-661-g0a941e5-1bionic [132 kB] 2020-08-24T00:40:52.179 INFO:teuthology.orchestra.run.smithi060.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.4-661-g0a941e5-1bionic [26.5 kB] 2020-08-24T00:40:52.180 INFO:teuthology.orchestra.run.smithi060.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.4-661-g0a941e5-1bionic [312 kB] 2020-08-24T00:40:52.189 INFO:teuthology.orchestra.run.smithi060.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.4-661-g0a941e5-1bionic [5,398 kB] 2020-08-24T00:40:52.261 INFO:teuthology.orchestra.run.smithi060.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2020-08-24T00:40:52.261 INFO:teuthology.orchestra.run.smithi060.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2020-08-24T00:40:52.265 INFO:teuthology.orchestra.run.smithi060.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2020-08-24T00:40:52.368 INFO:teuthology.orchestra.run.smithi060.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.4-661-g0a941e5-1bionic [99.2 kB] 2020-08-24T00:40:52.370 INFO:teuthology.orchestra.run.smithi060.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.4-661-g0a941e5-1bionic [335 kB] 2020-08-24T00:40:52.380 INFO:teuthology.orchestra.run.smithi060.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2020-08-24T00:40:52.381 INFO:teuthology.orchestra.run.smithi060.stdout:Get:18 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.4-661-g0a941e5-1bionic [18.7 MB] 2020-08-24T00:40:52.383 INFO:teuthology.orchestra.run.smithi060.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2020-08-24T00:40:52.426 INFO:teuthology.orchestra.run.smithi060.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2020-08-24T00:40:52.433 INFO:teuthology.orchestra.run.smithi060.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2020-08-24T00:40:52.442 INFO:teuthology.orchestra.run.smithi060.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2020-08-24T00:40:52.451 INFO:teuthology.orchestra.run.smithi060.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2020-08-24T00:40:52.460 INFO:teuthology.orchestra.run.smithi060.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.3 [221 kB] 2020-08-24T00:40:52.484 INFO:teuthology.orchestra.run.smithi060.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2020-08-24T00:40:52.500 INFO:teuthology.orchestra.run.smithi060.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2020-08-24T00:40:52.505 INFO:teuthology.orchestra.run.smithi060.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2020-08-24T00:40:52.520 INFO:teuthology.orchestra.run.smithi060.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2020-08-24T00:40:52.580 INFO:teuthology.orchestra.run.smithi060.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2020-08-24T00:40:52.580 INFO:teuthology.orchestra.run.smithi060.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2020-08-24T00:40:52.581 INFO:teuthology.orchestra.run.smithi060.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2020-08-24T00:40:52.584 INFO:teuthology.orchestra.run.smithi060.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2020-08-24T00:40:52.585 INFO:teuthology.orchestra.run.smithi060.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2020-08-24T00:40:52.585 INFO:teuthology.orchestra.run.smithi060.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2020-08-24T00:40:52.588 INFO:teuthology.orchestra.run.smithi060.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2020-08-24T00:40:52.659 INFO:teuthology.orchestra.run.smithi060.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2020-08-24T00:40:52.660 INFO:teuthology.orchestra.run.smithi060.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2020-08-24T00:40:52.661 INFO:teuthology.orchestra.run.smithi060.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2020-08-24T00:40:52.738 INFO:teuthology.orchestra.run.smithi060.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2020-08-24T00:40:52.738 INFO:teuthology.orchestra.run.smithi060.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2020-08-24T00:40:52.739 INFO:teuthology.orchestra.run.smithi060.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2020-08-24T00:40:52.741 INFO:teuthology.orchestra.run.smithi060.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2020-08-24T00:40:52.744 INFO:teuthology.orchestra.run.smithi060.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1 [174 kB] 2020-08-24T00:40:52.751 INFO:teuthology.orchestra.run.smithi060.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2020-08-24T00:40:52.755 INFO:teuthology.orchestra.run.smithi060.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2020-08-24T00:40:52.817 INFO:teuthology.orchestra.run.smithi060.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2020-08-24T00:40:52.818 INFO:teuthology.orchestra.run.smithi060.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2020-08-24T00:40:52.906 INFO:teuthology.orchestra.run.smithi060.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2020-08-24T00:40:52.907 INFO:teuthology.orchestra.run.smithi060.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2020-08-24T00:40:52.996 INFO:teuthology.orchestra.run.smithi060.stdout:Get:50 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.4-661-g0a941e5-1bionic [5,152 kB] 2020-08-24T00:40:53.164 INFO:teuthology.orchestra.run.smithi060.stdout:Get:51 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.4-661-g0a941e5-1bionic [147 kB] 2020-08-24T00:40:53.168 INFO:teuthology.orchestra.run.smithi060.stdout:Get:52 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.4-661-g0a941e5-1bionic [1,280 kB] 2020-08-24T00:40:53.189 INFO:teuthology.orchestra.run.smithi060.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2020-08-24T00:40:53.210 INFO:teuthology.orchestra.run.smithi060.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.4-661-g0a941e5-1bionic [5,943 kB] 2020-08-24T00:40:53.229 INFO:teuthology.orchestra.run.smithi060.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2020-08-24T00:40:53.230 INFO:teuthology.orchestra.run.smithi060.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2020-08-24T00:40:53.271 INFO:teuthology.orchestra.run.smithi060.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2020-08-24T00:40:53.271 INFO:teuthology.orchestra.run.smithi060.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2020-08-24T00:40:53.272 INFO:teuthology.orchestra.run.smithi060.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.2 [28.6 kB] 2020-08-24T00:40:53.272 INFO:teuthology.orchestra.run.smithi060.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2020-08-24T00:40:53.273 INFO:teuthology.orchestra.run.smithi060.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2020-08-24T00:40:53.273 INFO:teuthology.orchestra.run.smithi060.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2020-08-24T00:40:53.275 INFO:teuthology.orchestra.run.smithi060.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2020-08-24T00:40:53.347 INFO:teuthology.orchestra.run.smithi060.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2020-08-24T00:40:53.349 INFO:teuthology.orchestra.run.smithi060.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2020-08-24T00:40:53.416 INFO:teuthology.orchestra.run.smithi060.stdout:Get:66 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.4-661-g0a941e5-1bionic [22.7 MB] 2020-08-24T00:40:53.426 INFO:teuthology.orchestra.run.smithi060.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2020-08-24T00:40:53.428 INFO:teuthology.orchestra.run.smithi060.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2020-08-24T00:40:53.433 INFO:teuthology.orchestra.run.smithi060.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2020-08-24T00:40:53.434 INFO:teuthology.orchestra.run.smithi060.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2020-08-24T00:40:53.435 INFO:teuthology.orchestra.run.smithi060.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2020-08-24T00:40:53.439 INFO:teuthology.orchestra.run.smithi060.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2020-08-24T00:40:53.441 INFO:teuthology.orchestra.run.smithi060.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2020-08-24T00:40:53.442 INFO:teuthology.orchestra.run.smithi060.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2020-08-24T00:40:53.505 INFO:teuthology.orchestra.run.smithi060.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2020-08-24T00:40:53.508 INFO:teuthology.orchestra.run.smithi060.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2020-08-24T00:40:53.585 INFO:teuthology.orchestra.run.smithi060.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2020-08-24T00:40:53.586 INFO:teuthology.orchestra.run.smithi060.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2020-08-24T00:40:53.587 INFO:teuthology.orchestra.run.smithi060.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2020-08-24T00:40:53.588 INFO:teuthology.orchestra.run.smithi060.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2020-08-24T00:40:53.590 INFO:teuthology.orchestra.run.smithi060.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2020-08-24T00:40:53.594 INFO:teuthology.orchestra.run.smithi060.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2020-08-24T00:40:53.595 INFO:teuthology.orchestra.run.smithi060.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2020-08-24T00:40:53.596 INFO:teuthology.orchestra.run.smithi060.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2020-08-24T00:40:53.664 INFO:teuthology.orchestra.run.smithi060.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2020-08-24T00:40:53.666 INFO:teuthology.orchestra.run.smithi060.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.1 [1,091 kB] 2020-08-24T00:40:53.743 INFO:teuthology.orchestra.run.smithi060.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2020-08-24T00:40:53.744 INFO:teuthology.orchestra.run.smithi060.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2020-08-24T00:40:53.746 INFO:teuthology.orchestra.run.smithi060.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2020-08-24T00:40:53.838 INFO:teuthology.orchestra.run.smithi060.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2020-08-24T00:40:53.839 INFO:teuthology.orchestra.run.smithi060.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2020-08-24T00:40:53.839 INFO:teuthology.orchestra.run.smithi060.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.3 [330 kB] 2020-08-24T00:40:53.842 INFO:teuthology.orchestra.run.smithi060.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2020-08-24T00:40:53.843 INFO:teuthology.orchestra.run.smithi060.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2020-08-24T00:40:53.843 INFO:teuthology.orchestra.run.smithi060.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2020-08-24T00:40:53.844 INFO:teuthology.orchestra.run.smithi060.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2020-08-24T00:40:53.901 INFO:teuthology.orchestra.run.smithi060.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2020-08-24T00:40:53.902 INFO:teuthology.orchestra.run.smithi060.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2020-08-24T00:40:54.233 INFO:teuthology.orchestra.run.smithi060.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.4-661-g0a941e5-1bionic [3,860 B] 2020-08-24T00:40:54.234 INFO:teuthology.orchestra.run.smithi060.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.4-661-g0a941e5-1bionic [514 kB] 2020-08-24T00:40:54.239 INFO:teuthology.orchestra.run.smithi060.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.4-661-g0a941e5-1bionic [1,847 kB] 2020-08-24T00:40:54.258 INFO:teuthology.orchestra.run.smithi060.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.4-661-g0a941e5-1bionic [44.7 kB] 2020-08-24T00:40:54.259 INFO:teuthology.orchestra.run.smithi060.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.4-661-g0a941e5-1bionic [51.0 kB] 2020-08-24T00:40:54.260 INFO:teuthology.orchestra.run.smithi060.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.4-661-g0a941e5-1bionic [3,163 kB] 2020-08-24T00:40:54.346 INFO:teuthology.orchestra.run.smithi060.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.4-661-g0a941e5-1bionic [34.2 kB] 2020-08-24T00:40:54.346 INFO:teuthology.orchestra.run.smithi060.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.4-661-g0a941e5-1bionic [7,886 kB] 2020-08-24T00:40:54.606 INFO:teuthology.orchestra.run.smithi060.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.4-661-g0a941e5-1bionic [3,976 B] 2020-08-24T00:40:54.606 INFO:teuthology.orchestra.run.smithi060.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.4-661-g0a941e5-1bionic [22.4 kB] 2020-08-24T00:40:54.607 INFO:teuthology.orchestra.run.smithi060.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.4-661-g0a941e5-1bionic [43.7 MB] 2020-08-24T00:40:56.043 INFO:teuthology.orchestra.run.smithi060.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.4-661-g0a941e5-1bionic [17.9 kB] 2020-08-24T00:40:56.044 INFO:teuthology.orchestra.run.smithi060.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.4-661-g0a941e5-1bionic [8,567 kB] 2020-08-24T00:40:56.321 INFO:teuthology.orchestra.run.smithi060.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/octopus/0a941e55043ccee6d35a8717bec55ff7581e67d8/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.4-661-g0a941e5-1bionic [62.4 kB] 2020-08-24T00:40:57.040 INFO:teuthology.orchestra.run.smithi060.stdout:Fetched 157 MB in 5s (34.4 MB/s) 2020-08-24T00:40:57.183 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-uritemplate. 2020-08-24T00:40:57.220 INFO:teuthology.orchestra.run.smithi060.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102055 files and directories currently installed.) 2020-08-24T00:40:57.221 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2020-08-24T00:40:57.265 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2020-08-24T00:40:57.585 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2020-08-24T00:40:57.596 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2020-08-24T00:40:57.677 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2020-08-24T00:40:57.992 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../002-librbd1_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:40:58.206 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librbd1 (15.2.4-661-g0a941e5-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-08-24T00:40:58.736 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../003-librados2_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:40:58.860 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librados2 (15.2.4-661-g0a941e5-1bionic) over (12.2.13-0ubuntu0.18.04.2) ... 2020-08-24T00:40:59.447 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs2. 2020-08-24T00:40:59.453 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../004-libcephfs2_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:40:59.597 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs2 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:40:59.883 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rados. 2020-08-24T00:40:59.893 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../005-python3-rados_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:40:59.933 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rados (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:00.270 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-argparse. 2020-08-24T00:41:00.279 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:00.319 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-argparse (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:00.580 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cephfs. 2020-08-24T00:41:00.593 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../007-python3-cephfs_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:00.622 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cephfs (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:00.891 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-ceph-common. 2020-08-24T00:41:00.901 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:00.933 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-ceph-common (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:01.260 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-prettytable. 2020-08-24T00:41:01.272 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2020-08-24T00:41:01.311 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2020-08-24T00:41:01.563 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rbd. 2020-08-24T00:41:01.574 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../010-python3-rbd_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:01.622 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rbd (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:01.941 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librabbitmq4:amd64. 2020-08-24T00:41:01.952 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2020-08-24T00:41:01.983 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-08-24T00:41:02.269 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librdkafka1:amd64. 2020-08-24T00:41:02.280 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2020-08-24T00:41:02.319 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2020-08-24T00:41:02.655 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package librgw2. 2020-08-24T00:41:02.667 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../013-librgw2_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:02.718 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking librgw2 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:03.507 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rgw. 2020-08-24T00:41:03.520 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../014-python3-rgw_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:03.549 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rgw (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:03.918 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2020-08-24T00:41:03.930 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2020-08-24T00:41:03.960 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2020-08-24T00:41:04.262 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2020-08-24T00:41:04.275 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2020-08-24T00:41:04.313 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2020-08-24T00:41:04.641 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package liboath0. 2020-08-24T00:41:04.652 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2020-08-24T00:41:04.682 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking liboath0 (2.6.1-1) ... 2020-08-24T00:41:04.935 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libradosstriper1. 2020-08-24T00:41:04.947 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../018-libradosstriper1_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:04.976 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libradosstriper1 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:05.405 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-common. 2020-08-24T00:41:05.416 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../019-ceph-common_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:05.454 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-common (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:07.168 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-base. 2020-08-24T00:41:07.180 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../020-ceph-base_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:07.219 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-base (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:08.132 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-dateutil. 2020-08-24T00:41:08.143 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2020-08-24T00:41:08.181 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2020-08-24T00:41:08.509 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-asn1crypto. 2020-08-24T00:41:08.521 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2020-08-24T00:41:08.559 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2020-08-24T00:41:08.828 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cffi-backend. 2020-08-24T00:41:08.844 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2020-08-24T00:41:08.879 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2020-08-24T00:41:09.165 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cryptography. 2020-08-24T00:41:09.177 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.3_amd64.deb ... 2020-08-24T00:41:09.206 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-08-24T00:41:09.540 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-openssl. 2020-08-24T00:41:09.545 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2020-08-24T00:41:09.577 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2020-08-24T00:41:09.838 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2020-08-24T00:41:09.850 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:09.888 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-modules-core (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:10.209 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bcrypt. 2020-08-24T00:41:10.221 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2020-08-24T00:41:10.259 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2020-08-24T00:41:10.536 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cherrypy3. 2020-08-24T00:41:10.548 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2020-08-24T00:41:10.586 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2020-08-24T00:41:10.923 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jwt. 2020-08-24T00:41:10.935 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2020-08-24T00:41:10.998 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2020-08-24T00:41:11.301 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-logutils. 2020-08-24T00:41:11.312 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2020-08-24T00:41:11.351 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-logutils (0.3.3-5) ... 2020-08-24T00:41:11.654 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-markupsafe. 2020-08-24T00:41:11.666 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2020-08-24T00:41:11.704 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2020-08-24T00:41:11.998 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-mako. 2020-08-24T00:41:12.010 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2020-08-24T00:41:12.048 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2020-08-24T00:41:12.368 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplegeneric. 2020-08-24T00:41:12.380 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2020-08-24T00:41:12.418 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2020-08-24T00:41:12.721 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-singledispatch. 2020-08-24T00:41:12.732 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2020-08-24T00:41:12.771 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2020-08-24T00:41:13.073 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webob. 2020-08-24T00:41:13.085 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2020-08-24T00:41:13.123 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2020-08-24T00:41:13.401 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-bs4. 2020-08-24T00:41:13.413 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2020-08-24T00:41:13.442 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2020-08-24T00:41:13.704 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-waitress. 2020-08-24T00:41:13.717 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2020-08-24T00:41:13.754 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-waitress (1.0.1-1) ... 2020-08-24T00:41:14.057 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tempita. 2020-08-24T00:41:14.068 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2020-08-24T00:41:14.107 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tempita (0.5.2-2) ... 2020-08-24T00:41:14.418 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-paste. 2020-08-24T00:41:14.430 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2020-08-24T00:41:14.468 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-08-24T00:41:14.796 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2020-08-24T00:41:14.808 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2020-08-24T00:41:14.846 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2020-08-24T00:41:15.157 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastedeploy. 2020-08-24T00:41:15.168 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2020-08-24T00:41:15.207 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2020-08-24T00:41:15.510 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webtest. 2020-08-24T00:41:15.527 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2020-08-24T00:41:15.560 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2020-08-24T00:41:15.896 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pecan. 2020-08-24T00:41:15.909 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2020-08-24T00:41:15.946 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pecan (1.2.1-2) ... 2020-08-24T00:41:16.277 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-werkzeug. 2020-08-24T00:41:16.288 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1_all.deb ... 2020-08-24T00:41:16.318 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1) ... 2020-08-24T00:41:16.628 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr. 2020-08-24T00:41:16.635 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../045-ceph-mgr_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:16.673 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:17.124 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mon. 2020-08-24T00:41:17.146 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../046-ceph-mon_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:17.173 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mon (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:18.002 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-osd. 2020-08-24T00:41:18.014 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../047-ceph-osd_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:18.046 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-osd (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:19.950 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph. 2020-08-24T00:41:19.960 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../048-ceph_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:19.991 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:20.244 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-fuse. 2020-08-24T00:41:20.257 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../049-ceph-fuse_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:20.285 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-fuse (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:20.655 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mds. 2020-08-24T00:41:20.667 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../050-ceph-mds_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:20.746 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mds (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:21.334 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package cephadm. 2020-08-24T00:41:21.346 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../051-cephadm_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:21.384 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking cephadm (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:21.653 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jinja2. 2020-08-24T00:41:21.665 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2020-08-24T00:41:21.695 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-08-24T00:41:21.964 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2020-08-24T00:41:21.977 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:22.006 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-cephadm (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:22.284 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-repoze.lru. 2020-08-24T00:41:22.295 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2020-08-24T00:41:22.334 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2020-08-24T00:41:22.637 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-routes. 2020-08-24T00:41:22.654 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2020-08-24T00:41:22.689 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-routes (2.4.1-1) ... 2020-08-24T00:41:22.965 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2020-08-24T00:41:22.977 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:23.023 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-dashboard (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:23.872 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2020-08-24T00:41:23.886 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:23.914 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:24.225 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-numpy. 2020-08-24T00:41:24.238 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2020-08-24T00:41:24.334 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2020-08-24T00:41:25.681 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-decorator. 2020-08-24T00:41:25.692 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2020-08-24T00:41:25.731 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-decorator (4.1.2-1) ... 2020-08-24T00:41:26.044 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-scipy. 2020-08-24T00:41:26.055 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2020-08-24T00:41:26.094 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2020-08-24T00:41:27.214 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn-lib. 2020-08-24T00:41:27.227 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2020-08-24T00:41:27.264 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2020-08-24T00:41:27.726 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-joblib. 2020-08-24T00:41:27.737 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2020-08-24T00:41:27.776 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-joblib (0.11-1) ... 2020-08-24T00:41:28.121 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sklearn. 2020-08-24T00:41:28.133 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2020-08-24T00:41:28.170 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2020-08-24T00:41:28.702 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2020-08-24T00:41:28.713 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:28.753 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:29.897 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-monotonic. 2020-08-24T00:41:29.903 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2020-08-24T00:41:29.938 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-monotonic (1.1-2) ... 2020-08-24T00:41:30.216 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-fasteners. 2020-08-24T00:41:30.227 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2020-08-24T00:41:30.266 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2020-08-24T00:41:30.595 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-httplib2. 2020-08-24T00:41:30.605 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.2_all.deb ... 2020-08-24T00:41:30.644 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-08-24T00:41:30.947 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1. 2020-08-24T00:41:30.960 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2020-08-24T00:41:30.998 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2020-08-24T00:41:31.338 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyasn1-modules. 2020-08-24T00:41:31.346 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2020-08-24T00:41:31.375 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2020-08-24T00:41:31.649 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-rsa. 2020-08-24T00:41:31.661 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2020-08-24T00:41:31.695 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-rsa (3.4.2-1) ... 2020-08-24T00:41:31.990 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-sqlalchemy. 2020-08-24T00:41:32.007 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2020-08-24T00:41:32.048 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-08-24T00:41:32.486 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-oauth2client. 2020-08-24T00:41:32.499 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2020-08-24T00:41:32.536 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2020-08-24T00:41:32.874 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-setuptools. 2020-08-24T00:41:32.884 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2020-08-24T00:41:32.923 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2020-08-24T00:41:33.309 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-websocket. 2020-08-24T00:41:33.320 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2020-08-24T00:41:33.359 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2020-08-24T00:41:33.654 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-kubernetes. 2020-08-24T00:41:33.667 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2020-08-24T00:41:33.704 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2020-08-24T00:41:34.100 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2020-08-24T00:41:34.113 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:34.150 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-k8sevents (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:34.473 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-json-pointer. 2020-08-24T00:41:34.485 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2020-08-24T00:41:34.519 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-json-pointer (1.10-1) ... 2020-08-24T00:41:34.873 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-jsonpatch. 2020-08-24T00:41:34.886 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2020-08-24T00:41:34.926 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-08-24T00:41:35.217 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-mgr-rook. 2020-08-24T00:41:35.230 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.4-661-g0a941e5-1bionic_all.deb ... 2020-08-24T00:41:35.270 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-mgr-rook (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:35.604 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libonig4:amd64. 2020-08-24T00:41:35.616 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2020-08-24T00:41:35.721 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2020-08-24T00:41:36.074 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libjq1:amd64. 2020-08-24T00:41:36.086 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2020-08-24T00:41:36.123 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2020-08-24T00:41:36.393 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package jq. 2020-08-24T00:41:36.406 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2020-08-24T00:41:36.443 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking jq (1.5+dfsg-2) ... 2020-08-24T00:41:36.671 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package socat. 2020-08-24T00:41:36.683 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2020-08-24T00:41:36.721 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2020-08-24T00:41:37.083 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libxslt1.1:amd64. 2020-08-24T00:41:37.096 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2020-08-24T00:41:37.141 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-08-24T00:41:37.410 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package xmlstarlet. 2020-08-24T00:41:37.424 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2020-08-24T00:41:37.470 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2020-08-24T00:41:37.798 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package ceph-test. 2020-08-24T00:41:37.811 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../086-ceph-test_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:37.848 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking ceph-test (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:41.277 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package libcephfs-dev. 2020-08-24T00:41:45.045 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:45.046 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking libcephfs-dev (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:45.046 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-attr. 2020-08-24T00:41:45.046 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../088-python-attr_17.4.0-2_all.deb ... 2020-08-24T00:41:45.046 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-attr (17.4.0-2) ... 2020-08-24T00:41:45.047 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-funcsigs. 2020-08-24T00:41:45.047 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../089-python-funcsigs_1.0.2-4_all.deb ... 2020-08-24T00:41:45.047 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2020-08-24T00:41:45.047 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pluggy. 2020-08-24T00:41:45.047 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../090-python-pluggy_0.6.0-1_all.deb ... 2020-08-24T00:41:45.048 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pluggy (0.6.0-1) ... 2020-08-24T00:41:45.048 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-py. 2020-08-24T00:41:45.048 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../091-python-py_1.5.2-1_all.deb ... 2020-08-24T00:41:45.048 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-py (1.5.2-1) ... 2020-08-24T00:41:45.049 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python-pytest. 2020-08-24T00:41:45.049 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../092-python-pytest_3.3.2-2_all.deb ... 2020-08-24T00:41:45.049 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python-pytest (3.3.2-2) ... 2020-08-24T00:41:45.049 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-attr. 2020-08-24T00:41:45.049 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../093-python3-attr_17.4.0-2_all.deb ... 2020-08-24T00:41:45.055 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-attr (17.4.0-2) ... 2020-08-24T00:41:45.055 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-cycler. 2020-08-24T00:41:45.055 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../094-python3-cycler_0.10.0-1_all.deb ... 2020-08-24T00:41:45.055 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-cycler (0.10.0-1) ... 2020-08-24T00:41:45.056 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-webencodings. 2020-08-24T00:41:45.056 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../095-python3-webencodings_0.5-2_all.deb ... 2020-08-24T00:41:45.056 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-webencodings (0.5-2) ... 2020-08-24T00:41:45.056 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-html5lib. 2020-08-24T00:41:45.056 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../096-python3-html5lib_0.999999999-1_all.deb ... 2020-08-24T00:41:45.057 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2020-08-24T00:41:45.057 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-lxml:amd64. 2020-08-24T00:41:45.057 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../097-python3-lxml_4.2.1-1ubuntu0.1_amd64.deb ... 2020-08-24T00:41:45.057 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-08-24T00:41:45.169 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyparsing. 2020-08-24T00:41:45.183 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../098-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2020-08-24T00:41:45.228 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2020-08-24T00:41:45.514 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tz. 2020-08-24T00:41:45.528 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../099-python3-tz_2018.3-2_all.deb ... 2020-08-24T00:41:45.564 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tz (2018.3-2) ... 2020-08-24T00:41:45.858 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-matplotlib. 2020-08-24T00:41:45.871 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../100-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2020-08-24T00:41:45.920 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2020-08-24T00:41:46.598 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-olefile. 2020-08-24T00:41:46.612 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../101-python3-olefile_0.45.1-1_all.deb ... 2020-08-24T00:41:46.655 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-olefile (0.45.1-1) ... 2020-08-24T00:41:46.948 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pastescript. 2020-08-24T00:41:46.957 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../102-python3-pastescript_2.0.2-2_all.deb ... 2020-08-24T00:41:47.000 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2020-08-24T00:41:47.354 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pil:amd64. 2020-08-24T00:41:47.367 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../103-python3-pil_5.1.0-1ubuntu0.3_amd64.deb ... 2020-08-24T00:41:47.404 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-08-24T00:41:47.757 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pluggy. 2020-08-24T00:41:47.770 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../104-python3-pluggy_0.6.0-1_all.deb ... 2020-08-24T00:41:47.807 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2020-08-24T00:41:48.093 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-py. 2020-08-24T00:41:48.111 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../105-python3-py_1.5.2-1_all.deb ... 2020-08-24T00:41:48.151 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-py (1.5.2-1) ... 2020-08-24T00:41:48.463 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pyinotify. 2020-08-24T00:41:48.475 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../106-python3-pyinotify_0.9.6-1_all.deb ... 2020-08-24T00:41:48.505 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2020-08-24T00:41:48.741 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-pytest. 2020-08-24T00:41:48.752 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../107-python3-pytest_3.3.2-2_all.deb ... 2020-08-24T00:41:48.786 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-pytest (3.3.2-2) ... 2020-08-24T00:41:49.103 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-simplejson. 2020-08-24T00:41:49.121 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../108-python3-simplejson_3.13.2-1_amd64.deb ... 2020-08-24T00:41:49.161 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2020-08-24T00:41:49.464 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package python3-tk:amd64. 2020-08-24T00:41:49.476 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../109-python3-tk_3.6.9-1~18.04_amd64.deb ... 2020-08-24T00:41:49.509 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2020-08-24T00:41:49.985 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package radosgw. 2020-08-24T00:41:49.999 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../110-radosgw_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:50.035 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking radosgw (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:50.981 INFO:teuthology.orchestra.run.smithi060.stdout:Selecting previously unselected package rbd-fuse. 2020-08-24T00:41:50.986 INFO:teuthology.orchestra.run.smithi060.stdout:Preparing to unpack .../111-rbd-fuse_15.2.4-661-g0a941e5-1bionic_amd64.deb ... 2020-08-24T00:41:51.022 INFO:teuthology.orchestra.run.smithi060.stdout:Unpacking rbd-fuse (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:51.306 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-logutils (0.3.3-5) ... 2020-08-24T00:41:51.602 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1) ... 2020-08-24T00:41:51.982 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2020-08-24T00:41:52.104 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2020-08-24T00:41:52.732 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.3) ... 2020-08-24T00:41:53.126 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2020-08-24T00:41:53.426 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webencodings (0.5-2) ... 2020-08-24T00:41:53.719 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-attr (17.4.0-2) ... 2020-08-24T00:41:53.921 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2020-08-24T00:41:54.046 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-py (1.5.2-1) ... 2020-08-24T00:41:54.374 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-waitress (1.0.1-1) ... 2020-08-24T00:41:54.541 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2020-08-24T00:41:54.826 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-olefile (0.45.1-1) ... 2020-08-24T00:41:55.120 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-argparse (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:41:55.512 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tempita (0.5.2-2) ... 2020-08-24T00:41:55.824 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2020-08-24T00:41:55.949 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2020-08-24T00:41:56.901 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2020-08-24T00:41:57.219 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2020-08-24T00:41:57.512 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-py (1.5.2-1) ... 2020-08-24T00:41:57.747 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-simplejson (3.13.2-1) ... 2020-08-24T00:41:58.081 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-json-pointer (1.10-1) ... 2020-08-24T00:41:58.154 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2020-08-24T00:41:58.383 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2020-08-24T00:41:58.509 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pluggy (0.6.0-1) ... 2020-08-24T00:41:58.723 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2020-08-24T00:41:59.014 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2020-08-24T00:41:59.329 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cycler (0.10.0-1) ... 2020-08-24T00:41:59.639 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2020-08-24T00:42:00.001 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-bs4 (4.6.0-1) ... 2020-08-24T00:42:00.325 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pluggy (0.6.0-1) ... 2020-08-24T00:42:00.634 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up cephadm (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:01.328 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user cephadm....done 2020-08-24T00:42:01.475 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2020-08-24T00:42:01.768 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2020-08-24T00:42:01.894 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2020-08-24T00:42:02.229 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-prettytable (0.7.2-3) ... 2020-08-24T00:42:02.547 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2020-08-24T00:42:02.673 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2020-08-24T00:42:03.008 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-ceph-common (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:03.309 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-monotonic (1.1-2) ... 2020-08-24T00:42:03.602 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2020-08-24T00:42:03.737 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2020-08-24T00:42:04.088 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-html5lib (0.999999999-1) ... 2020-08-24T00:42:04.462 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2020-08-24T00:42:04.555 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2020-08-24T00:42:04.600 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2020-08-24T00:42:04.833 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-repoze.lru (0.7-1) ... 2020-08-24T00:42:05.135 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.2) ... 2020-08-24T00:42:05.444 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2020-08-24T00:42:05.789 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-attr (17.4.0-2) ... 2020-08-24T00:42:06.107 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-funcsigs (1.0.2-4) ... 2020-08-24T00:42:06.300 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2020-08-24T00:42:06.701 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2020-08-24T00:42:06.791 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2020-08-24T00:42:07.054 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liboath0 (2.6.1-1) ... 2020-08-24T00:42:07.229 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2020-08-24T00:42:07.356 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2020-08-24T00:42:07.607 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2020-08-24T00:42:07.717 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-setuptools (39.0.1-2) ... 2020-08-24T00:42:08.101 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2020-08-24T00:42:08.227 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-decorator (4.1.2-1) ... 2020-08-24T00:42:08.503 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-tz (2018.3-2) ... 2020-08-24T00:42:08.780 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2020-08-24T00:42:09.056 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2020-08-24T00:42:09.374 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-joblib (0.11-1) ... 2020-08-24T00:42:09.751 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-routes (2.4.1-1) ... 2020-08-24T00:42:10.060 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2020-08-24T00:42:10.378 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-dateutil (2.6.1-1) ... 2020-08-24T00:42:10.771 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.3) ... 2020-08-24T00:42:11.114 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-fasteners (0.12.0-3) ... 2020-08-24T00:42:11.399 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up jq (1.5+dfsg-2) ... 2020-08-24T00:42:11.508 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2020-08-24T00:42:12.707 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.1) ... 2020-08-24T00:42:12.998 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2020-08-24T00:42:13.308 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2020-08-24T00:42:13.435 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pytest (3.3.2-2) ... 2020-08-24T00:42:13.810 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python-pytest (3.3.2-2) ... 2020-08-24T00:42:14.081 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2020-08-24T00:42:14.179 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up xmlstarlet (1.6.1-2) ... 2020-08-24T00:42:14.314 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librados2 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:14.415 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rsa (3.4.2-1) ... 2020-08-24T00:42:14.724 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs2 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:14.845 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libcephfs-dev (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:14.968 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rados (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:15.095 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2020-08-24T00:42:15.420 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2020-08-24T00:42:15.696 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2020-08-24T00:42:16.018 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2020-08-24T00:42:16.868 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-sklearn (0.19.1-3) ... 2020-08-24T00:42:17.789 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2020-08-24T00:42:18.094 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up libradosstriper1 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:18.220 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-cephfs (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:18.538 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librgw2 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:18.656 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-fuse (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:18.863 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-08-24T00:42:18.863 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2020-08-24T00:42:19.140 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up librbd1 (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:19.242 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pecan (1.2.1-2) ... 2020-08-24T00:42:19.338 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2020-08-24T00:42:19.358 INFO:teuthology.orchestra.run.smithi060.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2020-08-24T00:42:19.685 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2020-08-24T00:42:20.374 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-modules-core (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:20.493 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-pastescript (2.0.2-2) ... 2020-08-24T00:42:20.828 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rgw (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:20.946 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up rbd-fuse (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:21.051 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up python3-rbd (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:21.177 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-common (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:21.662 INFO:teuthology.orchestra.run.smithi060.stdout:Adding group ceph....done 2020-08-24T00:42:22.031 INFO:teuthology.orchestra.run.smithi060.stdout:Adding system user ceph....done 2020-08-24T00:42:22.224 INFO:teuthology.orchestra.run.smithi060.stdout:Setting system user ceph properties....done 2020-08-24T00:42:22.381 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2020-08-24T00:42:22.624 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2020-08-24T00:42:22.938 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up radosgw (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:23.176 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-08-24T00:42:23.177 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2020-08-24T00:42:23.557 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-test (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:23.658 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-base (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:23.989 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2020-08-24T00:42:24.268 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:24.480 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-08-24T00:42:24.480 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2020-08-24T00:42:24.778 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-local (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:24.921 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-osd (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:25.367 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-08-24T00:42:25.368 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2020-08-24T00:42:25.657 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-cephadm (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:25.767 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:25.877 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-rook (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:26.402 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mds (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:26.625 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-08-24T00:42:26.625 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2020-08-24T00:42:26.895 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mon (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:27.114 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-08-24T00:42:27.114 INFO:teuthology.orchestra.run.smithi060.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2020-08-24T00:42:27.408 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-k8sevents (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:27.506 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph-mgr-dashboard (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:27.632 INFO:teuthology.orchestra.run.smithi060.stdout:Setting up ceph (15.2.4-661-g0a941e5-1bionic) ... 2020-08-24T00:42:27.734 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for systemd (237-3ubuntu10.42) ... 2020-08-24T00:42:27.918 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2020-08-24T00:42:29.006 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2020-08-24T00:42:29.047 INFO:teuthology.orchestra.run.smithi060.stdout:ureadahead will be reprofiled on next reboot 2020-08-24T00:42:29.105 INFO:teuthology.orchestra.run.smithi060.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.2) ... 2020-08-24T00:42:29.549 INFO:teuthology.orchestra.run.smithi060.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2020-08-24T00:42:29.558 DEBUG:teuthology.parallel:result is None 2020-08-24T00:42:29.564 INFO:teuthology.orchestra.run.smithi060:> sudo lsb_release -is 2020-08-24T00:42:29.630 INFO:teuthology.orchestra.run.smithi060.stdout:Ubuntu 2020-08-24T00:42:29.634 DEBUG:teuthology.misc:System to be installed: Ubuntu 2020-08-24T00:42:29.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=0a941e55043ccee6d35a8717bec55ff7581e67d8 2020-08-24T00:42:29.839 INFO:teuthology.orchestra.run.smithi060:> dpkg-query -W -f '${Version}' ceph 2020-08-24T00:42:29.869 INFO:teuthology.orchestra.run.smithi060.stdout:15.2.4-661-g0a941e5-1bionic 2020-08-24T00:42:29.870 INFO:teuthology.packaging:The installed version of ceph is 15.2.4-661-g0a941e5-1bionic 2020-08-24T00:42:29.871 INFO:teuthology.task.install:The correct ceph version 15.2.4-661-g0a941e5-1bionic is installed. 2020-08-24T00:42:29.871 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-08-24T00:42:29.872 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-08-24T00:42:29.928 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-08-24T00:42:29.929 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-08-24T00:42:29.986 INFO:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-08-24T00:42:30.044 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-08-24T00:42:30.044 INFO:teuthology.orchestra.run.smithi060:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-08-24T00:42:30.133 INFO:teuthology.orchestra.run.smithi060:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-08-24T00:42:30.193 INFO:teuthology.run_tasks:Running task workunit... 2020-08-24T00:42:30.251 INFO:tasks.workunit:Pulling workunits from ref 0a941e55043ccee6d35a8717bec55ff7581e67d8 2020-08-24T00:42:30.254 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-08-24T00:42:30.254 INFO:tasks.workunit:timeout=3h 2020-08-24T00:42:30.254 INFO:tasks.workunit:cleanup=True 2020-08-24T00:42:30.255 INFO:teuthology.orchestra.run.smithi060:> stat -- /home/ubuntu/cephtest/mnt.0 2020-08-24T00:42:30.265 INFO:teuthology.orchestra.run.smithi060.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-08-24T00:42:30.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-08-24T00:42:30.267 INFO:teuthology.orchestra.run.smithi060:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-08-24T00:42:30.312 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-08-24T00:42:30.313 INFO:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-08-24T00:42:30.362 INFO:teuthology.orchestra.run.smithi060:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 0a941e55043ccee6d35a8717bec55ff7581e67d8 2020-08-24T00:42:30.469 INFO:tasks.workunit.client.0.smithi060.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-08-24T00:51:05.384 INFO:tasks.workunit.client.0.smithi060.stderr:Note: checking out '0a941e55043ccee6d35a8717bec55ff7581e67d8'. 2020-08-24T00:51:05.385 INFO:tasks.workunit.client.0.smithi060.stderr: 2020-08-24T00:51:05.386 INFO:tasks.workunit.client.0.smithi060.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-08-24T00:51:05.386 INFO:tasks.workunit.client.0.smithi060.stderr:changes and commit them, and you can discard any commits you make in this 2020-08-24T00:51:05.386 INFO:tasks.workunit.client.0.smithi060.stderr:state without impacting any branches by performing another checkout. 2020-08-24T00:51:05.386 INFO:tasks.workunit.client.0.smithi060.stderr: 2020-08-24T00:51:05.386 INFO:tasks.workunit.client.0.smithi060.stderr:If you want to create a new branch to retain commits you create, you may 2020-08-24T00:51:05.387 INFO:tasks.workunit.client.0.smithi060.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-08-24T00:51:05.387 INFO:tasks.workunit.client.0.smithi060.stderr: 2020-08-24T00:51:05.387 INFO:tasks.workunit.client.0.smithi060.stderr: git checkout -b 2020-08-24T00:51:05.387 INFO:tasks.workunit.client.0.smithi060.stderr: 2020-08-24T00:51:05.387 INFO:tasks.workunit.client.0.smithi060.stderr:HEAD is now at 0a941e5504 Merge pull request #36655 from smithfarm/wip-46968-octopus 2020-08-24T00:51:05.389 INFO:teuthology.orchestra.run.smithi060:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-08-24T00:51:05.465 DEBUG:teuthology.orchestra.remote:smithi060:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-08-24T00:51:05.476 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-08-24T00:51:05.477 INFO:tasks.workunit:Running workunit osd/bad-inc-map.sh... 2020-08-24T00:51:05.477 INFO:teuthology.orchestra.run.smithi060:workunit test osd/bad-inc-map.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a941e55043ccee6d35a8717bec55ff7581e67d8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh 2020-08-24T00:51:05.496 INFO:tasks.workunit.client.0.smithi060.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-08-24T00:51:05.541 INFO:tasks.workunit.client.0.smithi060.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-08-24T00:51:05.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T00:51:05.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T00:51:05.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PYTHONWARNINGS=ignore 2020-08-24T00:51:05.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PYTHONWARNINGS=ignore 2020-08-24T00:51:05.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export CEPH_CONF=/dev/null 2020-08-24T00:51:05.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: CEPH_CONF=/dev/null 2020-08-24T00:51:05.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: unset CEPH_ARGS 2020-08-24T00:51:05.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: local code 2020-08-24T00:51:05.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: run td/bad-inc-map 2020-08-24T00:51:05.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:8: run: local dir=td/bad-inc-map 2020-08-24T00:51:05.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:9: run: shift 2020-08-24T00:51:05.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: export CEPH_MON=127.0.0.1:46153 2020-08-24T00:51:05.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:11: run: CEPH_MON=127.0.0.1:46153 2020-08-24T00:51:05.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:12: run: export CEPH_ARGS 2020-08-24T00:51:05.545 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: uuidgen 2020-08-24T00:51:05.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:13: run: CEPH_ARGS+='--fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none ' 2020-08-24T00:51:05.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:14: run: CEPH_ARGS+='--mon-host=127.0.0.1:46153 ' 2020-08-24T00:51:05.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:15: run: set -e 2020-08-24T00:51:05.550 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: set 2020-08-24T00:51:05.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-08-24T00:51:05.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:17: run: local funcs=TEST_bad_inc_map 2020-08-24T00:51:05.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:18: run: for func in $funcs 2020-08-24T00:51:05.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:19: run: setup td/bad-inc-map 2020-08-24T00:51:05.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/bad-inc-map 2020-08-24T00:51:05.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/bad-inc-map 2020-08-24T00:51:05.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2020-08-24T00:51:05.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T00:51:05.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2020-08-24T00:51:05.558 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:51:05.558 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:51:05.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:51:05.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:51:05.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:51:05.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:51:05.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:51:05.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:51:05.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:51:05.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:51:05.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:51:05.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:51:05.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:51:05.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:51:05.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:51:05.564 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:51:05.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:51:05.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T00:51:05.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2020-08-24T00:51:05.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:51:05.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:05.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:05.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.10182 2020-08-24T00:51:05.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:51:05.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:51:05.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/bad-inc-map 2020-08-24T00:51:05.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T00:51:05.571 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:05.571 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:05.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.10182 2020-08-24T00:51:05.573 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T00:51:05.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-08-24T00:51:05.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-08-24T00:51:05.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T00:51:05.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/bad-inc-map 1' TERM HUP INT 2020-08-24T00:51:05.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:20: run: TEST_bad_inc_map td/bad-inc-map 2020-08-24T00:51:05.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:26: TEST_bad_inc_map: local dir=td/bad-inc-map 2020-08-24T00:51:05.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:28: TEST_bad_inc_map: run_mon td/bad-inc-map a 2020-08-24T00:51:05.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/bad-inc-map 2020-08-24T00:51:05.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T00:51:05.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T00:51:05.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T00:51:05.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/bad-inc-map/a 2020-08-24T00:51:05.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/bad-inc-map/a --run-dir=td/bad-inc-map 2020-08-24T00:51:06.020 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T00:51:06.021 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:51:06.021 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:51:06.021 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:51:06.021 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:06.021 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:06.022 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:06.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/bad-inc-map/a '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --mon-cluster-log-file=td/bad-inc-map/log --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T00:51:06.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T00:51:06.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T00:51:06.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:51:06.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:51:06.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T00:51:06.200 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T00:51:06.200 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:51:06.200 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:51:06.200 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:51:06.201 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:51:06.201 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:06.201 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:06.202 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.10182/ceph-mon.a.asok 2020-08-24T00:51:06.202 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:51:06.202 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.10182/ceph-mon.a.asok config get fsid 2020-08-24T00:51:06.650 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T00:51:06.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:51:06.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:51:06.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T00:51:06.652 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T00:51:06.653 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:51:06.653 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:51:06.653 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:51:06.653 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:51:06.653 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:06.654 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:06.654 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.10182/ceph-mon.a.asok 2020-08-24T00:51:06.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:51:06.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.10182/ceph-mon.a.asok config get mon_host 2020-08-24T00:51:06.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:29: TEST_bad_inc_map: run_mgr td/bad-inc-map x 2020-08-24T00:51:06.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/bad-inc-map 2020-08-24T00:51:06.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T00:51:06.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T00:51:06.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T00:51:06.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/bad-inc-map/x 2020-08-24T00:51:06.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T00:51:07.073 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T00:51:07.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:51:07.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:51:07.074 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:51:07.074 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:07.074 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:07.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:07.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:51:07.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/bad-inc-map/x '--log-file=td/bad-inc-map/$name.log' '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --run-dir=td/bad-inc-map '--pid-file=td/bad-inc-map/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:51:07.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:30: TEST_bad_inc_map: run_osd td/bad-inc-map 0 2020-08-24T00:51:07.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/bad-inc-map 2020-08-24T00:51:07.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:51:07.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T00:51:07.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:51:07.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/bad-inc-map/0 2020-08-24T00:51:07.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 ' 2020-08-24T00:51:07.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:51:07.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:51:07.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:51:07.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/0' 2020-08-24T00:51:07.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/0/journal' 2020-08-24T00:51:07.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:51:07.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:51:07.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2020-08-24T00:51:07.137 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:51:07.137 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:51:07.138 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:51:07.138 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:51:07.138 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:07.138 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:07.138 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:07.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:07.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:51:07.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:51:07.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:51:07.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2020-08-24T00:51:07.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2020-08-24T00:51:07.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:51:07.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:51:07.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:51:07.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:51:07.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:51:07.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/bad-inc-map/0 2020-08-24T00:51:07.143 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:51:07.143 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=edfcf145-73e5-4237-a6d8-e783098780c8 2020-08-24T00:51:07.143 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 edfcf145-73e5-4237-a6d8-e783098780c8' 2020-08-24T00:51:07.143 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:51:07.143 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 edfcf145-73e5-4237-a6d8-e783098780c8 2020-08-24T00:51:07.154 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD7DkNf8DEYCRAA/wj8/cA23w8s4Pc6TdQjYQ== 2020-08-24T00:51:07.154 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD7DkNf8DEYCRAA/wj8/cA23w8s4Pc6TdQjYQ=="}' 2020-08-24T00:51:07.155 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new edfcf145-73e5-4237-a6d8-e783098780c8 -i td/bad-inc-map/0/new.json 2020-08-24T00:51:07.482 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:51:07.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/bad-inc-map/0/new.json 2020-08-24T00:51:07.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD7DkNf8DEYCRAA/wj8/cA23w8s4Pc6TdQjYQ== --osd-uuid edfcf145-73e5-4237-a6d8-e783098780c8 2020-08-24T00:51:07.522 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:07.515+0000 7f9b82d8bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:07.524 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:07.519+0000 7f9b82d8bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:07.525 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:07.519+0000 7f9b82d8bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:07.526 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:07.519+0000 7f9b82d8bd80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2020-08-24T00:51:07.526 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:07.519+0000 7f9b82d8bd80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2020-08-24T00:51:07.526 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:07.519+0000 7f9b82d8bd80 -1 bluestore(td/bad-inc-map/0/block) _read_bdev_label failed to open td/bad-inc-map/0/block: (2) No such file or directory 2020-08-24T00:51:07.584 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:07.579+0000 7f9b82d8bd80 -1 bluestore(td/bad-inc-map/0) _read_fsid unparsable uuid 2020-08-24T00:51:08.114 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:08.111+0000 7f9b82d8bd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:51:10.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/bad-inc-map/0/keyring 2020-08-24T00:51:10.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:51:10.977 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T00:51:10.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T00:51:10.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/bad-inc-map/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:51:11.247 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T00:51:11.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T00:51:11.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:51:11.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/0 --osd-journal=td/bad-inc-map/0/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:51:11.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:51:11.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:51:11.273 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:11.271+0000 7f4295f14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:11.287 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:11.283+0000 7f4295f14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:11.287 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:11.283+0000 7f4295f14d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:11.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T00:51:11.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:51:11.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:51:11.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:51:11.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:51:11.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:51:11.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:51:11.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:11.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:51:11.479 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:51:11.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:51:11.830 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:11.827+0000 7f4295f14d80 -1 Falling back to public interface 2020-08-24T00:51:12.614 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:12.611+0000 7f4295f14d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T00:51:12.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:51:12.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:51:12.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:51:12.682 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:51:12.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:12.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:51:13.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:51:14.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:51:14.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:51:14.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T00:51:14.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:14.033 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T00:51:14.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:51:14.403 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/10547,v1:127.0.0.1:6801/10547] [v2:127.0.0.1:6802/10547,v1:127.0.0.1:6803/10547] exists,up edfcf145-73e5-4237-a6d8-e783098780c8 2020-08-24T00:51:14.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:51:14.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:51:14.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:51:14.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:31: TEST_bad_inc_map: run_osd td/bad-inc-map 1 2020-08-24T00:51:14.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/bad-inc-map 2020-08-24T00:51:14.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:51:14.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T00:51:14.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:51:14.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/bad-inc-map/1 2020-08-24T00:51:14.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 ' 2020-08-24T00:51:14.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:51:14.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:51:14.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:51:14.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/1' 2020-08-24T00:51:14.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/1/journal' 2020-08-24T00:51:14.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:51:14.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:51:14.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2020-08-24T00:51:14.408 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:51:14.409 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:51:14.409 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:51:14.411 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:51:14.411 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:14.411 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:14.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:14.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:14.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:51:14.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:51:14.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:51:14.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2020-08-24T00:51:14.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2020-08-24T00:51:14.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:51:14.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:51:14.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:51:14.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:51:14.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:51:14.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/bad-inc-map/1 2020-08-24T00:51:14.414 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:51:14.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c7c0d6e2-7267-4728-a8ee-0c80c36064a8 2020-08-24T00:51:14.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 c7c0d6e2-7267-4728-a8ee-0c80c36064a8' 2020-08-24T00:51:14.415 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:51:14.416 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 c7c0d6e2-7267-4728-a8ee-0c80c36064a8 2020-08-24T00:51:14.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQACD0Nfqs0LGRAAH0nK492Ul21DjUazyzHQMg== 2020-08-24T00:51:14.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQACD0Nfqs0LGRAAH0nK492Ul21DjUazyzHQMg=="}' 2020-08-24T00:51:14.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c7c0d6e2-7267-4728-a8ee-0c80c36064a8 -i td/bad-inc-map/1/new.json 2020-08-24T00:51:14.901 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:51:14.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/bad-inc-map/1/new.json 2020-08-24T00:51:14.917 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQACD0Nfqs0LGRAAH0nK492Ul21DjUazyzHQMg== --osd-uuid c7c0d6e2-7267-4728-a8ee-0c80c36064a8 2020-08-24T00:51:14.947 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:14.943+0000 7fa0fd997d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:14.950 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:14.947+0000 7fa0fd997d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:14.951 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:14.947+0000 7fa0fd997d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:14.952 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:14.951+0000 7fa0fd997d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2020-08-24T00:51:14.952 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:14.951+0000 7fa0fd997d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2020-08-24T00:51:14.953 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:14.951+0000 7fa0fd997d80 -1 bluestore(td/bad-inc-map/1/block) _read_bdev_label failed to open td/bad-inc-map/1/block: (2) No such file or directory 2020-08-24T00:51:15.002 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:14.999+0000 7fa0fd997d80 -1 bluestore(td/bad-inc-map/1) _read_fsid unparsable uuid 2020-08-24T00:51:15.384 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:15.379+0000 7fa0fd997d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:51:18.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/bad-inc-map/1/keyring 2020-08-24T00:51:18.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:51:18.043 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T00:51:18.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T00:51:18.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/bad-inc-map/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:51:18.505 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:51:18.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T00:51:18.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/1 --osd-journal=td/bad-inc-map/1/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:51:18.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:51:18.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:51:18.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:51:18.528 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:18.523+0000 7fa6d53b3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:18.530 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:18.527+0000 7fa6d53b3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:18.531 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:18.527+0000 7fa6d53b3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:18.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T00:51:18.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:51:18.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:51:18.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:51:18.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:51:18.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:51:18.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:51:18.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:18.857 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:51:18.857 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:51:19.070 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:19.067+0000 7fa6d53b3d80 -1 Falling back to public interface 2020-08-24T00:51:19.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:51:19.854 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:19.851+0000 7fa6d53b3d80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T00:51:20.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:51:20.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:51:20.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:51:20.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:20.183 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:51:20.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:51:20.568 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/10990,v1:127.0.0.1:6811/10990] [v2:127.0.0.1:6812/10990,v1:127.0.0.1:6813/10990] exists,up c7c0d6e2-7267-4728-a8ee-0c80c36064a8 2020-08-24T00:51:20.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:51:20.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:51:20.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:51:20.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:32: TEST_bad_inc_map: run_osd td/bad-inc-map 2 2020-08-24T00:51:20.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/bad-inc-map 2020-08-24T00:51:20.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:51:20.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T00:51:20.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:51:20.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/bad-inc-map/2 2020-08-24T00:51:20.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 ' 2020-08-24T00:51:20.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:51:20.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:51:20.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:51:20.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/bad-inc-map/2' 2020-08-24T00:51:20.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/bad-inc-map/2/journal' 2020-08-24T00:51:20.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:51:20.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:51:20.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/bad-inc-map' 2020-08-24T00:51:20.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:51:20.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:51:20.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:51:20.583 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:51:20.583 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:51:20.584 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:51:20.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:20.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' 2020-08-24T00:51:20.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:51:20.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:51:20.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:51:20.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/bad-inc-map/$name.log' 2020-08-24T00:51:20.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/bad-inc-map/$name.pid' 2020-08-24T00:51:20.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:51:20.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:51:20.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:51:20.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:51:20.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:51:20.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/bad-inc-map/2 2020-08-24T00:51:20.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:51:20.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=46b9dfc8-7308-4423-a0d0-c06d4544f387 2020-08-24T00:51:20.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 46b9dfc8-7308-4423-a0d0-c06d4544f387' 2020-08-24T00:51:20.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:51:20.590 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 46b9dfc8-7308-4423-a0d0-c06d4544f387 2020-08-24T00:51:20.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAID0NfazxwIxAAiv0B3JEiXtMRz6wrrebzvg== 2020-08-24T00:51:20.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAID0NfazxwIxAAiv0B3JEiXtMRz6wrrebzvg=="}' 2020-08-24T00:51:20.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 46b9dfc8-7308-4423-a0d0-c06d4544f387 -i td/bad-inc-map/2/new.json 2020-08-24T00:51:21.138 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T00:51:21.150 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/bad-inc-map/2/new.json 2020-08-24T00:51:21.151 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAID0NfazxwIxAAiv0B3JEiXtMRz6wrrebzvg== --osd-uuid 46b9dfc8-7308-4423-a0d0-c06d4544f387 2020-08-24T00:51:21.181 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.179+0000 7f152503dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:21.208 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.203+0000 7f152503dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:21.208 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.203+0000 7f152503dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:21.208 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.203+0000 7f152503dd80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2020-08-24T00:51:21.209 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.203+0000 7f152503dd80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2020-08-24T00:51:21.209 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.203+0000 7f152503dd80 -1 bluestore(td/bad-inc-map/2/block) _read_bdev_label failed to open td/bad-inc-map/2/block: (2) No such file or directory 2020-08-24T00:51:21.304 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.303+0000 7f152503dd80 -1 bluestore(td/bad-inc-map/2) _read_fsid unparsable uuid 2020-08-24T00:51:21.720 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:21.715+0000 7f152503dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:51:24.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/bad-inc-map/2/keyring 2020-08-24T00:51:24.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:51:24.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T00:51:24.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/bad-inc-map/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:51:24.419 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T00:51:24.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T00:51:24.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:51:24.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=eee891cb-367d-40f6-9ba4-35443e7eb7c5 --auth-supported=none --mon-host=127.0.0.1:46153 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/bad-inc-map/2 --osd-journal=td/bad-inc-map/2/journal --chdir= --run-dir=td/bad-inc-map '--admin-socket=/tmp/ceph-asok.10182/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/bad-inc-map/$name.log' '--pid-file=td/bad-inc-map/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:51:24.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:51:24.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:51:24.877 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T00:51:24.898 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:24.895+0000 7fef8a72ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:24.900 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:24.899+0000 7fef8a72ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:24.901 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:24.899+0000 7fef8a72ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:51:25.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T00:51:25.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:51:25.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:51:25.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:51:25.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:51:25.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:51:25.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:51:25.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:25.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:51:25.217 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:51:25.442 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:25.439+0000 7fef8a72ad80 -1 Falling back to public interface 2020-08-24T00:51:25.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:51:26.354 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:51:26.351+0000 7fef8a72ad80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T00:51:26.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:51:26.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:51:26.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:51:26.575 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:51:26.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:26.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:51:27.045 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/11345,v1:127.0.0.1:6819/11345] [v2:127.0.0.1:6820/11345,v1:127.0.0.1:6821/11345] exists,up 46b9dfc8-7308-4423-a0d0-c06d4544f387 2020-08-24T00:51:27.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:51:27.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:51:27.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:51:27.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:34: TEST_bad_inc_map: ceph config set osd.2 osd_inject_bad_map_crc_probability 1 2020-08-24T00:51:27.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:37: TEST_bad_inc_map: create_pool foo 8 2020-08-24T00:51:27.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create foo 8 2020-08-24T00:51:28.036 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'foo' already exists 2020-08-24T00:51:28.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T00:51:29.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:38: TEST_bad_inc_map: ceph osd pool set foo min_size 1 2020-08-24T00:51:29.800 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 min_size to 1 2020-08-24T00:51:29.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:39: TEST_bad_inc_map: ceph osd pool set foo min_size 2 2020-08-24T00:51:30.862 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 min_size to 2 2020-08-24T00:51:30.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:41: TEST_bad_inc_map: sleep 5 2020-08-24T00:51:35.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: TIMEOUT=10 2020-08-24T00:51:35.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:44: TEST_bad_inc_map: wait_for_osd up 0 2020-08-24T00:51:35.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:51:35.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:51:35.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:51:35.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:51:35.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 10 )) 2020-08-24T00:51:35.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:51:35.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:35.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:51:35.870 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:51:36.230 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/10547,v1:127.0.0.1:6801/10547] [v2:127.0.0.1:6802/10547,v1:127.0.0.1:6803/10547] exists,up edfcf145-73e5-4237-a6d8-e783098780c8 2020-08-24T00:51:36.230 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:51:36.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:51:36.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:51:36.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:51:36.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: TIMEOUT=10 2020-08-24T00:51:36.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:45: TEST_bad_inc_map: wait_for_osd up 1 2020-08-24T00:51:36.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:51:36.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:51:36.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:51:36.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:51:36.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 10 )) 2020-08-24T00:51:36.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:51:36.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:51:36.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:51:36.815 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/10990,v1:127.0.0.1:6811/10990] [v2:127.0.0.1:6812/10990,v1:127.0.0.1:6813/10990] exists,up c7c0d6e2-7267-4728-a8ee-0c80c36064a8 2020-08-24T00:52:09.732 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:52:09.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:52:09.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:52:09.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:52:09.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: TIMEOUT=10 2020-08-24T00:52:09.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:46: TEST_bad_inc_map: wait_for_osd up 2 2020-08-24T00:52:09.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:52:09.738 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:52:09.738 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:52:09.738 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:52:09.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 10 )) 2020-08-24T00:52:09.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:52:09.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:52:09.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:52:09.766 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 25 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/11345,v1:127.0.0.1:6819/11345] [v2:127.0.0.1:6820/11345,v1:127.0.0.1:6821/11345] exists,up 46b9dfc8-7308-4423-a0d0-c06d4544f387 2020-08-24T00:52:09.766 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:27.919+0000 7fef77586700 -1 osd.2 15 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.767 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:28.355+0000 7fef77586700 -1 osd.2 17 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.767 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:28.523+0000 7fef77586700 -1 osd.2 18 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.767 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:29.551+0000 7fef72379700 -1 osd.2 19 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.767 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:29.943+0000 7fef72379700 -1 osd.2 22 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.768 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.071+0000 7fef77586700 -1 osd.2 23 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.768 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.503+0000 7fef77586700 -1 osd.2 24 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.768 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.847+0000 7fef77586700 -1 osd.2 25 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.769 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.851+0000 7fef77586700 -1 osd.2 25 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.769 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:31.323+0000 7fef77586700 -1 osd.2 26 handle_osd_map injecting map crc failure 2020-08-24T00:52:09.769 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:27.919+0000 7fef77586700 10 osd.2 15 handle_osd_map bailing because last < start (15<16) 2020-08-24T00:52:09.769 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:28.355+0000 7fef77586700 10 osd.2 17 handle_osd_map bailing because last < start (17<18) 2020-08-24T00:52:09.770 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:28.523+0000 7fef77586700 10 osd.2 18 handle_osd_map bailing because last < start (18<19) 2020-08-24T00:52:09.771 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:29.551+0000 7fef72379700 10 osd.2 19 handle_osd_map bailing because last < start (19<20) 2020-08-24T00:52:09.771 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:29.947+0000 7fef72379700 10 osd.2 22 handle_osd_map bailing because last < start (22<23) 2020-08-24T00:52:09.771 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.071+0000 7fef77586700 10 osd.2 23 handle_osd_map bailing because last < start (23<24) 2020-08-24T00:52:09.772 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.503+0000 7fef77586700 10 osd.2 24 handle_osd_map bailing because last < start (24<25) 2020-08-24T00:52:09.772 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.847+0000 7fef77586700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2020-08-24T00:52:09.772 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:30.851+0000 7fef77586700 10 osd.2 25 handle_osd_map bailing because last < start (25<26) 2020-08-24T00:52:09.773 INFO:tasks.workunit.client.0.smithi060.stdout:2020-08-24T00:51:31.323+0000 7fef77586700 10 osd.2 26 handle_osd_map bailing because last < start (26<27) 2020-08-24T00:52:09.773 INFO:tasks.workunit.client.0.smithi060.stdout:success 2020-08-24T00:52:09.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:52:09.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:52:09.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:52:09.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:49: TEST_bad_inc_map: grep 'injecting map crc failure' td/bad-inc-map/osd.2.log 2020-08-24T00:52:09.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:50: TEST_bad_inc_map: grep 'bailing because last' td/bad-inc-map/osd.2.log 2020-08-24T00:52:09.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:52: TEST_bad_inc_map: echo success 2020-08-24T00:52:09.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:54: TEST_bad_inc_map: delete_pool foo 2020-08-24T00:52:09.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=foo 2020-08-24T00:52:09.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete foo foo --yes-i-really-really-mean-it 2020-08-24T00:52:09.777 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'foo' does not exist 2020-08-24T00:52:09.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:55: TEST_bad_inc_map: kill_daemons td/bad-inc-map 2020-08-24T00:52:09.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:52:09.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:52:09.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:52:09.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:52:09.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:52:09.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:52:09.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/bad-inc-map.sh:21: run: teardown td/bad-inc-map 2020-08-24T00:52:09.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2020-08-24T00:52:09.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T00:52:09.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2020-08-24T00:52:09.781 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:52:09.781 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:52:09.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:52:09.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:52:09.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:52:09.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:52:09.783 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:52:09.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:52:09.783 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:52:09.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:52:09.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:52:09.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:52:09.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:52:09.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:52:09.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:52:09.785 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:52:09.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:52:09.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T00:52:09.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2020-08-24T00:52:09.787 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:52:09.787 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:09.787 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:52:09.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.10182 2020-08-24T00:52:09.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:52:09.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:52:09.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: code=0 2020-08-24T00:52:09.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: teardown td/bad-inc-map 0 2020-08-24T00:52:09.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/bad-inc-map 2020-08-24T00:52:09.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-08-24T00:52:09.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/bad-inc-map KILL 2020-08-24T00:52:09.791 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:52:09.791 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:52:09.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:52:09.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:52:09.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:52:09.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:52:09.793 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:52:09.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:52:09.793 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:52:09.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:52:09.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:52:09.794 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:52:09.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:52:09.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:52:09.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:52:09.795 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:52:09.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:52:09.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-08-24T00:52:09.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/bad-inc-map 2020-08-24T00:52:09.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:52:09.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:09.797 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.10182 2020-08-24T00:52:09.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.10182 2020-08-24T00:52:09.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:52:09.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:52:09.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: return 0 2020-08-24T00:52:09.800 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-08-24T00:52:09.800 INFO:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-08-24T00:52:09.819 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-08-24T00:52:09.820 INFO:teuthology.orchestra.run.smithi060:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a941e55043ccee6d35a8717bec55ff7581e67d8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-08-24T00:52:09.878 INFO:tasks.workunit.client.0.smithi060.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-08-24T00:52:09.892 INFO:tasks.workunit.client.0.smithi060.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-08-24T00:52:09.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T00:52:09.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T00:52:09.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PYTHONWARNINGS=ignore 2020-08-24T00:52:09.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PYTHONWARNINGS=ignore 2020-08-24T00:52:09.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export CEPH_CONF=/dev/null 2020-08-24T00:52:09.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: CEPH_CONF=/dev/null 2020-08-24T00:52:09.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: unset CEPH_ARGS 2020-08-24T00:52:09.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: local code 2020-08-24T00:52:09.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: run td/divergent-priors 2020-08-24T00:52:09.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-08-24T00:52:09.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-08-24T00:52:09.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-08-24T00:52:09.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-08-24T00:52:09.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-08-24T00:52:09.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-08-24T00:52:09.896 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-08-24T00:52:09.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-08-24T00:52:09.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-08-24T00:52:09.898 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-08-24T00:52:09.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-08-24T00:52:09.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-08-24T00:52:09.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-08-24T00:52:09.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-08-24T00:52:09.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-08-24T00:52:09.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-08-24T00:52:09.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-08-24T00:52:09.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-08-24T00:52:09.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-08-24T00:52:09.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-08-24T00:52:09.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-08-24T00:52:09.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-08-24T00:52:09.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-08-24T00:52:09.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-08-24T00:52:09.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-08-24T00:52:09.902 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-08-24T00:52:09.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none ' 2020-08-24T00:52:09.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-08-24T00:52:09.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-08-24T00:52:09.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-08-24T00:52:09.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:52:09.904 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-08-24T00:52:09.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-08-24T00:52:09.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-08-24T00:52:09.906 INFO:tasks.workunit.client.0.smithi060.stderr:TEST_divergent_2 2020-08-24T00:52:09.907 INFO:tasks.workunit.client.0.smithi060.stderr:TEST_divergent_3 2020-08-24T00:52:09.907 INFO:tasks.workunit.client.0.smithi060.stderr:TEST_divergent_ec' 2020-08-24T00:52:09.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-08-24T00:52:09.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-08-24T00:52:09.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-08-24T00:52:09.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-08-24T00:52:09.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T00:52:09.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T00:52:09.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T00:52:09.909 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:52:09.909 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:52:09.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:52:09.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:52:09.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:52:09.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:52:09.911 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:52:09.911 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:52:09.912 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:52:09.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:52:09.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:52:09.915 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:52:09.915 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:52:09.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:52:09.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:52:09.917 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:52:09.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:52:09.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T00:52:09.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T00:52:09.921 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:52:09.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:09.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:09.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T00:52:09.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:52:09.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:52:09.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-08-24T00:52:09.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T00:52:09.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:09.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:09.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.11819 2020-08-24T00:52:09.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T00:52:09.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-08-24T00:52:09.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-08-24T00:52:09.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T00:52:09.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-08-24T00:52:09.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-08-24T00:52:09.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-08-24T00:52:09.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-08-24T00:52:09.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-08-24T00:52:09.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-08-24T00:52:09.932 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-08-24T00:52:09.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-08-24T00:52:09.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-08-24T00:52:09.944 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:52:09.944 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:52:09.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-08-24T00:52:09.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-08-24T00:52:09.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T00:52:09.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T00:52:09.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T00:52:09.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-08-24T00:52:09.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-08-24T00:52:10.176 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T00:52:10.177 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:52:10.177 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:52:10.177 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:52:10.177 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:10.177 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:10.178 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:10.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T00:52:10.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T00:52:10.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T00:52:10.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:52:10.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:52:10.357 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T00:52:10.357 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T00:52:10.358 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:52:10.358 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:52:10.358 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:52:10.359 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:52:10.359 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:10.359 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:10.359 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T00:52:10.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:52:10.361 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get fsid 2020-08-24T00:52:10.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T00:52:10.459 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:52:10.459 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:52:10.459 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T00:52:10.461 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T00:52:10.461 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:52:10.461 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:52:10.461 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:52:10.462 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:52:10.462 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:10.462 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:10.462 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T00:52:10.462 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:52:10.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get mon_host 2020-08-24T00:52:10.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-08-24T00:52:10.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-08-24T00:52:10.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T00:52:10.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T00:52:10.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T00:52:10.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-08-24T00:52:10.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T00:52:11.069 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T00:52:11.069 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:52:11.069 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:52:11.071 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:52:11.071 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:11.071 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:11.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:11.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:52:11.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:52:11.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-08-24T00:52:11.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-08-24T00:52:11.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:52:11.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:52:11.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T00:52:11.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:52:11.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-08-24T00:52:11.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:52:11.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:52:11.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:52:11.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:52:11.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T00:52:11.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T00:52:11.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:52:11.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:52:11.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:52:11.132 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:52:11.133 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:52:11.133 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:52:11.133 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:52:11.133 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:11.134 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:11.134 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:11.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:11.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:52:11.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:52:11.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:52:11.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:52:11.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:52:11.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:52:11.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:52:11.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:52:11.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:52:11.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:52:11.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-08-24T00:52:11.137 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:52:11.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0013942e-a74c-45d5-a0d9-ab468c0875f7 2020-08-24T00:52:11.138 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 0013942e-a74c-45d5-a0d9-ab468c0875f7' 2020-08-24T00:52:11.138 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:52:11.138 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 0013942e-a74c-45d5-a0d9-ab468c0875f7 2020-08-24T00:52:11.139 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA7D0NfZHzgBxAAfMZq1a3jvFaFtQi/jQ8a9A== 2020-08-24T00:52:11.139 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA7D0NfZHzgBxAAfMZq1a3jvFaFtQi/jQ8a9A=="}' 2020-08-24T00:52:11.139 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0013942e-a74c-45d5-a0d9-ab468c0875f7 -i td/divergent-priors/0/new.json 2020-08-24T00:52:11.470 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:52:11.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-08-24T00:52:11.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA7D0NfZHzgBxAAfMZq1a3jvFaFtQi/jQ8a9A== --osd-uuid 0013942e-a74c-45d5-a0d9-ab468c0875f7 2020-08-24T00:52:11.521 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.514+0000 7f90e5a2dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:11.530 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.518+0000 7f90e5a2dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:11.531 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.522+0000 7f90e5a2dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:11.531 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.522+0000 7f90e5a2dd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:52:11.531 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.522+0000 7f90e5a2dd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:52:11.532 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.522+0000 7f90e5a2dd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:52:11.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.570+0000 7f90e5a2dd80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-08-24T00:52:11.978 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:11.970+0000 7f90e5a2dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:52:14.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-08-24T00:52:14.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:52:14.651 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T00:52:14.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T00:52:14.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:52:15.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T00:52:15.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:52:15.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:52:15.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:52:15.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:52:15.121 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T00:52:15.154 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:15.150+0000 7fe76bce8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:15.193 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:15.186+0000 7fe76bce8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:15.193 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:15.186+0000 7fe76bce8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:15.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T00:52:15.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:52:15.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:52:15.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:52:15.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:52:15.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:52:15.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:52:15.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:52:15.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:52:15.476 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:52:15.734 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:15.730+0000 7fe76bce8d80 -1 Falling back to public interface 2020-08-24T00:52:15.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:52:16.525 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:16.522+0000 7fe76bce8d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T00:52:16.840 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:52:16.840 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:52:16.841 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:52:16.841 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:52:16.842 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:52:16.842 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:52:17.234 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/12249,v1:127.0.0.1:6803/12249] [v2:127.0.0.1:6804/12249,v1:127.0.0.1:6805/12249] exists,up 0013942e-a74c-45d5-a0d9-ab468c0875f7 2020-08-24T00:52:17.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:52:17.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:52:17.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:52:17.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-08-24T00:52:17.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-08-24T00:52:17.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:52:17.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:52:17.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T00:52:17.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:52:17.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-08-24T00:52:17.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:52:17.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:52:17.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:52:17.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:52:17.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T00:52:17.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T00:52:17.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:52:17.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:52:17.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:52:17.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:52:17.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:52:17.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:52:17.239 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:52:17.239 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:17.239 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:17.239 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:17.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:17.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:52:17.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:52:17.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:52:17.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:52:17.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:52:17.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:52:17.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:52:17.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:52:17.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:52:17.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:52:17.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-08-24T00:52:17.245 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:52:17.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ed163f29-105f-4f32-a324-7646df6f5a35 2020-08-24T00:52:17.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 ed163f29-105f-4f32-a324-7646df6f5a35' 2020-08-24T00:52:17.245 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:52:17.245 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 ed163f29-105f-4f32-a324-7646df6f5a35 2020-08-24T00:52:17.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBBD0Nf5b1JDxAA5GG5cxKS4RGRGH21bBFB8Q== 2020-08-24T00:52:17.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBBD0Nf5b1JDxAA5GG5cxKS4RGRGH21bBFB8Q=="}' 2020-08-24T00:52:17.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ed163f29-105f-4f32-a324-7646df6f5a35 -i td/divergent-priors/1/new.json 2020-08-24T00:52:17.821 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:52:17.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-08-24T00:52:17.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBBD0Nf5b1JDxAA5GG5cxKS4RGRGH21bBFB8Q== --osd-uuid ed163f29-105f-4f32-a324-7646df6f5a35 2020-08-24T00:52:17.879 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:17.874+0000 7fbc9f66bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:17.895 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:17.890+0000 7fbc9f66bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:17.896 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:17.894+0000 7fbc9f66bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:17.897 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:17.894+0000 7fbc9f66bd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:52:17.897 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:17.894+0000 7fbc9f66bd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:52:17.898 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:17.894+0000 7fbc9f66bd80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:52:17.961 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:17.958+0000 7fbc9f66bd80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-08-24T00:52:18.382 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:18.378+0000 7fbc9f66bd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:52:21.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-08-24T00:52:21.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:52:21.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T00:52:21.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:52:21.061 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T00:52:21.530 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:52:21.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T00:52:21.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:52:21.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:52:21.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:52:21.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:52:21.565 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:21.562+0000 7f04114ffd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:21.567 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:21.562+0000 7f04114ffd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:21.568 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:21.566+0000 7f04114ffd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:21.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T00:52:21.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:52:21.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:52:21.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:52:21.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:52:21.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:52:21.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:52:21.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:52:21.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:52:21.884 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:52:22.102 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:22.098+0000 7f04114ffd80 -1 Falling back to public interface 2020-08-24T00:52:22.276 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:52:22.895 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:22.886+0000 7f04114ffd80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T00:52:23.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:52:23.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:52:23.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:52:23.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:52:23.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:52:23.280 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:52:23.685 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/12603,v1:127.0.0.1:6811/12603] [v2:127.0.0.1:6812/12603,v1:127.0.0.1:6813/12603] exists,up ed163f29-105f-4f32-a324-7646df6f5a35 2020-08-24T00:52:23.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:52:23.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:52:23.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:52:23.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-08-24T00:52:23.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-08-24T00:52:23.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:52:23.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:52:23.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T00:52:23.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:52:23.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-08-24T00:52:23.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:52:23.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:52:23.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:52:23.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:52:23.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T00:52:23.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T00:52:23.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:52:23.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:52:23.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:52:23.690 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:52:23.691 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:52:23.691 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:52:23.691 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:52:23.692 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:52:23.703 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:52:23.703 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:23.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:52:23.704 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:52:23.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:52:23.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:52:23.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:52:23.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:52:23.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:52:23.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:52:23.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:52:23.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:52:23.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:52:23.707 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-08-24T00:52:23.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:52:23.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2a35f1c2-02a9-48c5-b86c-a57c332d7bc5 2020-08-24T00:52:23.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 2a35f1c2-02a9-48c5-b86c-a57c332d7bc5' 2020-08-24T00:52:23.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:52:23.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBHD0NfweJFKhAArFt0pj4ZOW/u9XkSWqyiCw== 2020-08-24T00:52:23.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBHD0NfweJFKhAArFt0pj4ZOW/u9XkSWqyiCw=="}' 2020-08-24T00:52:23.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2a35f1c2-02a9-48c5-b86c-a57c332d7bc5 -i td/divergent-priors/2/new.json 2020-08-24T00:52:23.720 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 2a35f1c2-02a9-48c5-b86c-a57c332d7bc5 2020-08-24T00:52:24.396 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T00:52:24.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-08-24T00:52:24.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBHD0NfweJFKhAArFt0pj4ZOW/u9XkSWqyiCw== --osd-uuid 2a35f1c2-02a9-48c5-b86c-a57c332d7bc5 2020-08-24T00:52:24.439 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.434+0000 7fbed099dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:24.442 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.438+0000 7fbed099dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:24.443 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.438+0000 7fbed099dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:24.444 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.442+0000 7fbed099dd80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:52:24.444 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.442+0000 7fbed099dd80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:52:24.445 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.442+0000 7fbed099dd80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:52:24.479 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.474+0000 7fbed099dd80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-08-24T00:52:24.930 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:24.922+0000 7fbed099dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:52:27.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-08-24T00:52:27.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:52:27.550 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T00:52:27.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T00:52:27.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:52:28.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T00:52:28.028 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T00:52:28.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:52:28.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:52:28.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:52:28.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:52:28.065 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:28.062+0000 7f0a4bdc5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:28.076 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:28.066+0000 7f0a4bdc5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:28.076 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:28.066+0000 7f0a4bdc5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:52:28.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T00:52:28.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:52:28.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:52:28.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:52:28.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:52:28.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:52:28.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:52:28.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:52:28.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:52:28.378 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:52:28.602 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:28.598+0000 7f0a4bdc5d80 -1 Falling back to public interface 2020-08-24T00:52:28.760 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:52:29.411 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:52:29.406+0000 7f0a4bdc5d80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T00:52:29.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:52:29.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:52:29.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:52:29.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:52:29.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:52:29.763 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:52:30.135 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/12969,v1:127.0.0.1:6819/12969] [v2:127.0.0.1:6820/12969,v1:127.0.0.1:6821/12969] exists,up 2a35f1c2-02a9-48c5-b86c-a57c332d7bc5 2020-08-24T00:52:30.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:52:30.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:52:30.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:52:30.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-08-24T00:52:31.001 INFO:tasks.workunit.client.0.smithi060.stderr:noout is set 2020-08-24T00:52:31.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-08-24T00:52:31.850 INFO:tasks.workunit.client.0.smithi060.stderr:noin is set 2020-08-24T00:52:31.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-08-24T00:52:32.548 INFO:tasks.workunit.client.0.smithi060.stderr:nodown is set 2020-08-24T00:52:32.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-08-24T00:52:32.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-08-24T00:52:33.222 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T00:52:33.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T00:52:34.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-08-24T00:52:35.009 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 size to 3 2020-08-24T00:52:35.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-08-24T00:52:35.785 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 min_size to 2 2020-08-24T00:52:35.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-08-24T00:52:35.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:52:35.802 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:52:36.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:52:36.178 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:52:36.178 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:52:36.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:52:36.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:52:36.183 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:52:36.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-08-24T00:52:36.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836486 2020-08-24T00:52:36.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486' 2020-08-24T00:52:36.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:52:36.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:52:36.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-08-24T00:52:36.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672965 2020-08-24T00:52:36.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-08-24T00:52:36.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:52:36.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:52:36.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-08-24T00:52:36.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509444 2020-08-24T00:52:36.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-08-24T00:52:36.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:52:36.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836486 2020-08-24T00:52:36.687 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:52:36.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:52:36.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2020-08-24T00:52:36.689 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:52:36.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836486 2020-08-24T00:52:36.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-08-24T00:52:36.690 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:52:36.690 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836486 2020-08-24T00:52:37.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836486 -lt 21474836486 2020-08-24T00:52:37.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:52:37.046 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672965 2020-08-24T00:52:37.047 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:52:37.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:52:37.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-08-24T00:52:37.055 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:52:37.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672965 2020-08-24T00:52:37.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-08-24T00:52:37.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:52:37.056 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672965 2020-08-24T00:52:37.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672965 -lt 42949672965 2020-08-24T00:52:37.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:52:37.413 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509444 2020-08-24T00:52:37.414 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:52:37.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:52:37.418 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2020-08-24T00:52:37.418 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:52:37.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509444 2020-08-24T00:52:37.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-08-24T00:52:37.419 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509444 2020-08-24T00:52:37.420 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:52:37.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509442 -lt 64424509444 2020-08-24T00:52:37.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:52:38.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:53:54.348 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:53:54.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509444 -lt 64424509444 2020-08-24T00:53:54.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-08-24T00:53:54.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:53:54.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:53:54.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:53:54.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:53:54.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T00:53:54.350 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:53:54.351 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:53:54.351 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:53:54.351 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:53:54.351 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:53:54.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:53:54.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:53:54.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:53:54.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:53:54.352 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:53:54.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:53:54.353 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:53:54.353 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:53:54.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:53:54.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.353 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:53:54.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-08-24T00:53:54.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836489 2020-08-24T00:53:54.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489' 2020-08-24T00:53:54.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.354 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:53:54.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-08-24T00:53:54.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672968 2020-08-24T00:53:54.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-08-24T00:53:54.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.355 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:53:54.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-08-24T00:53:54.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509446 2020-08-24T00:53:54.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-08-24T00:53:54.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836489 2020-08-24T00:53:54.357 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:53:54.357 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2020-08-24T00:53:54.357 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836489 2020-08-24T00:53:54.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-08-24T00:53:54.358 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:53:54.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836487 -lt 21474836489 2020-08-24T00:53:54.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:53:54.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:53:54.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:53:54.359 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836489 -lt 21474836489 2020-08-24T00:53:54.359 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672968 2020-08-24T00:53:54.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:53:54.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2020-08-24T00:53:54.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672968 2020-08-24T00:53:54.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-08-24T00:53:54.361 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:53:54.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672968 -lt 42949672968 2020-08-24T00:53:54.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.361 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509446 2020-08-24T00:53:54.361 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:53:54.362 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2020-08-24T00:53:54.362 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509446 2020-08-24T00:53:54.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-08-24T00:53:54.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:53:54.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509447 -lt 64424509446 2020-08-24T00:53:54.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.364 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:53:54.364 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.364 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.364 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.364 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.365 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.365 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.365 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.365 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T00:53:54.366 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.367 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.367 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T00:53:54.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T00:53:54.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T00:53:54.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-08-24T00:53:54.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-08-24T00:53:54.370 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:53:54.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-08-24T00:53:54.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-08-24T00:53:54.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-08-24T00:53:54.371 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:53:54.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-08-24T00:53:54.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-08-24T00:53:54.372 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-08-24T00:53:54.372 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-08-24T00:53:54.372 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-08-24T00:53:54.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-08-24T00:53:54.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-08-24T00:53:54.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-08-24T00:53:54.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-08-24T00:53:54.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-08-24T00:53:54.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-08-24T00:53:54.374 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-08-24T00:53:54.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-08-24T00:53:54.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-08-24T00:53:54.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-08-24T00:53:54.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-08-24T00:53:54.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-08-24T00:53:54.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-08-24T00:53:54.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-08-24T00:53:54.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-08-24T00:53:54.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-08-24T00:53:54.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-08-24T00:53:54.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-08-24T00:53:54.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-08-24T00:53:54.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-08-24T00:53:54.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-08-24T00:53:54.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-08-24T00:53:54.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-08-24T00:53:54.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-08-24T00:53:54.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-08-24T00:53:54.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-08-24T00:53:54.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-08-24T00:53:54.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-08-24T00:53:54.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-08-24T00:53:54.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-08-24T00:53:54.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-08-24T00:53:54.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-08-24T00:53:54.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-08-24T00:53:54.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-08-24T00:53:54.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-08-24T00:53:54.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-08-24T00:53:54.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-08-24T00:53:54.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-08-24T00:53:54.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-08-24T00:53:54.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-08-24T00:53:54.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-08-24T00:53:54.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-08-24T00:53:54.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-08-24T00:53:54.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-08-24T00:53:54.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-08-24T00:53:54.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-08-24T00:53:54.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-08-24T00:53:54.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-08-24T00:53:54.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-08-24T00:53:54.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-08-24T00:53:54.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-08-24T00:53:54.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-08-24T00:53:54.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-08-24T00:53:54.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-08-24T00:53:54.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-08-24T00:53:54.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-08-24T00:53:54.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-08-24T00:53:54.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-08-24T00:53:54.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-08-24T00:53:54.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-08-24T00:53:54.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-08-24T00:53:54.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-08-24T00:53:54.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-08-24T00:53:54.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-08-24T00:53:54.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-08-24T00:53:54.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-08-24T00:53:54.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-08-24T00:53:54.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-08-24T00:53:54.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-08-24T00:53:54.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-08-24T00:53:54.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-08-24T00:53:54.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-08-24T00:53:54.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-08-24T00:53:54.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-08-24T00:53:54.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-08-24T00:53:54.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-08-24T00:53:54.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-08-24T00:53:54.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-08-24T00:53:54.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-08-24T00:53:54.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-08-24T00:53:54.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-08-24T00:53:54.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-08-24T00:53:54.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-08-24T00:53:54.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-08-24T00:53:54.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-08-24T00:53:54.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-08-24T00:53:54.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-08-24T00:53:54.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-08-24T00:53:54.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-08-24T00:53:54.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-08-24T00:53:54.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-08-24T00:53:54.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-08-24T00:53:54.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-08-24T00:53:54.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-08-24T00:53:54.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-08-24T00:53:54.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-08-24T00:53:54.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-08-24T00:53:54.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-08-24T00:53:54.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-08-24T00:53:54.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-08-24T00:53:54.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-08-24T00:53:54.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-08-24T00:53:54.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-08-24T00:53:54.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-08-24T00:53:54.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-08-24T00:53:54.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-08-24T00:53:54.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-08-24T00:53:54.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-08-24T00:53:54.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:53:54.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-08-24T00:53:54.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:53:54.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:53:54.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:53:54.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:53:54.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:53:54.428 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:53:54.428 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:53:54.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:53:54.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:53:54.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:53:54.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:53:54.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:53:54.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:53:54.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:53:54.431 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:53:54.431 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:53:54.431 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:53:54.431 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:53:54.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:53:54.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.432 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:53:54.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836495 2020-08-24T00:53:54.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836495 2020-08-24T00:53:54.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495' 2020-08-24T00:53:54.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.433 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:53:54.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672974 2020-08-24T00:53:54.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672974 2020-08-24T00:53:54.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2020-08-24T00:53:54.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.434 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:53:54.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509453 2020-08-24T00:53:54.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509453 2020-08-24T00:53:54.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509453' 2020-08-24T00:53:54.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836495 2020-08-24T00:53:54.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:53:54.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2020-08-24T00:53:54.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836495 2020-08-24T00:53:54.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-08-24T00:53:54.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:53:54.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836493 -lt 21474836495 2020-08-24T00:53:54.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:53:54.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:53:54.437 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:53:54.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836495 -lt 21474836495 2020-08-24T00:53:54.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672974 2020-08-24T00:53:54.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:53:54.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2020-08-24T00:53:54.439 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672974 2020-08-24T00:53:54.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2020-08-24T00:53:54.439 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:53:54.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672974 -lt 42949672974 2020-08-24T00:53:54.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509453 2020-08-24T00:53:54.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:53:54.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509453 2020-08-24T00:53:54.441 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509453 2020-08-24T00:53:54.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2020-08-24T00:53:54.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:53:54.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509453 -lt 64424509453 2020-08-24T00:53:54.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.443 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:53:54.443 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T00:53:54.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T00:53:54.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T00:53:54.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T00:53:54.446 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-08-24T00:53:54.446 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-08-24T00:53:54.446 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-08-24T00:53:54.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-08-24T00:53:54.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-08-24T00:53:54.447 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-08-24T00:53:54.447 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-08-24T00:53:54.447 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-08-24T00:53:54.447 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:53:54.448 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-08-24T00:53:54.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T00:53:54.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T00:53:54.448 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:53:54.449 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:53:54.449 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:53:54.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T00:53:54.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-08-24T00:53:54.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok config set objectstore_blackhole 1 2020-08-24T00:53:54.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-08-24T00:53:54.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T00:53:54.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T00:53:54.451 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:53:54.451 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:53:54.451 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:53:54.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T00:53:54.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-08-24T00:53:54.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok config set objectstore_blackhole 1 2020-08-24T00:53:54.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-08-24T00:53:54.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-08-24T00:53:54.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-08-24T00:53:54.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-08-24T00:53:54.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-08-24T00:53:54.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-08-24T00:53:54.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-08-24T00:53:54.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local pooltype=rep 2020-08-24T00:53:54.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-08-24T00:53:54.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local which=data 2020-08-24T00:53:54.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-08-24T00:53:54.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local poolname=test 2020-08-24T00:53:54.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-08-24T00:53:54.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local objname=existing_99 2020-08-24T00:53:54.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-08-24T00:53:54.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2238: inject_eio: local dir=td/divergent-priors 2020-08-24T00:53:54.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: shift 2020-08-24T00:53:54.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local shard_id=0 2020-08-24T00:53:54.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: shift 2020-08-24T00:53:54.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-08-24T00:53:54.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: get_osds test existing_99 2020-08-24T00:53:54.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-08-24T00:53:54.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-08-24T00:53:54.457 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-08-24T00:53:54.457 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-08-24T00:53:54.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-08-24T00:53:54.457 INFO:tasks.workunit.client.0.smithi060.stderr:0 2020-08-24T00:53:54.458 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:53:54.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-08-24T00:53:54.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2243: inject_eio: local -a initial_osds 2020-08-24T00:53:54.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: local osd_id=1 2020-08-24T00:53:54.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: '[' rep '!=' ec ']' 2020-08-24T00:53:54.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: shard_id= 2020-08-24T00:53:54.460 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: cat td/divergent-priors/1/type 2020-08-24T00:53:54.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: type=bluestore 2020-08-24T00:53:54.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2249: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-08-24T00:53:54.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-08-24T00:53:54.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-08-24T00:53:54.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-08-24T00:53:54.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-08-24T00:53:54.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-08-24T00:53:54.463 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-08-24T00:53:54.463 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T00:53:54.463 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T00:53:54.463 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:53:54.464 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:53:54.464 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:53:54.464 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T00:53:54.464 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.11819/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-08-24T00:53:54.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-08-24T00:53:54.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2250: inject_eio: local loop=0 2020-08-24T00:53:54.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2252: inject_eio: grep -q Invalid 2020-08-24T00:53:54.465 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: get_asok_path osd.1 2020-08-24T00:53:54.465 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T00:53:54.465 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T00:53:54.466 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:53:54.466 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:53:54.467 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:53:54.467 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T00:53:54.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: CEPH_ARGS= 2020-08-24T00:53:54.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2251: inject_eio: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.1.asok injectdataerr test existing_99 2020-08-24T00:53:54.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-08-24T00:53:54.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-08-24T00:53:54.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-08-24T00:53:54.468 INFO:tasks.workunit.client.0.smithi060.stderr:rados: no process found 2020-08-24T00:53:54.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-08-24T00:53:54.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-08-24T00:53:54.470 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:53:54.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-08-24T00:53:54.471 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:53:54.471 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:53:54.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:53:54.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:53:54.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:53:54.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:53:54.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-08-24T00:53:54.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-08-24T00:53:54.474 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already down. 2020-08-24T00:53:54.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-08-24T00:53:54.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-08-24T00:53:54.475 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already down. 2020-08-24T00:53:54.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-08-24T00:53:54.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-08-24T00:53:54.476 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already down. 2020-08-24T00:53:54.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-08-24T00:53:54.477 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already out. 2020-08-24T00:53:54.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-08-24T00:53:54.479 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already out. 2020-08-24T00:53:54.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-08-24T00:53:54.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-08-24T00:53:54.480 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:53:54.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-08-24T00:53:54.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:53:54.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:53:54.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-08-24T00:53:54.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:53:54.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-08-24T00:53:54.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:53:54.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:53:54.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:53:54.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:53:54.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T00:53:54.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T00:53:54.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:53:54.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:53:54.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:53:54.485 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:53:54.485 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:53:54.485 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:53:54.485 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:53:54.485 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:53:54.486 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:53:54.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:53:54.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:53:54.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:53:54.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:53:54.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:53:54.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:53:54.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:53:54.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:53:54.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:53:54.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:53:54.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-08-24T00:53:54.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-08-24T00:53:54.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:53:54.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-08-24T00:53:54.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-08-24T00:53:54.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:53:54.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:53:54.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:53:54.490 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:20.930+0000 7f5fbee2bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:53:54.490 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:20.934+0000 7f5fbee2bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:53:54.490 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:20.934+0000 7f5fbee2bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:53:54.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-08-24T00:53:54.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:53:54.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:53:54.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:53:54.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:53:54.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:53:54.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:53:54.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:53:54.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:53:54.492 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:21.466+0000 7f5fbee2bd80 -1 Falling back to public interface 2020-08-24T00:53:54.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:53:54.492 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:22.410+0000 7f5fbee2bd80 -1 osd.0 29 log_to_monitors {default=true} 2020-08-24T00:53:54.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:53:54.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:53:54.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:53:54.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:53:54.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:53:54.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:53:54.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:53:54.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:53:54.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-08-24T00:53:54.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:53:54.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:53:54.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-08-24T00:53:54.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:53:54.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-08-24T00:53:54.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:53:54.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:53:54.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:53:54.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:53:54.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T00:53:54.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T00:53:54.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:53:54.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:53:54.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:53:54.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:53:54.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:53:54.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:53:54.500 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:53:54.500 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:53:54.500 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:53:54.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:53:54.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:53:54.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:53:54.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:53:54.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:53:54.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:53:54.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:53:54.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:53:54.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:53:54.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:53:54.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-08-24T00:53:54.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-08-24T00:53:54.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:53:54.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-08-24T00:53:54.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-08-24T00:53:54.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:53:54.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:53:54.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:53:54.505 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:23.046+0000 7f92e9db1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:53:54.505 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:23.046+0000 7f92e9db1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:53:54.505 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:23.050+0000 7f92e9db1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:53:54.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-08-24T00:53:54.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:53:54.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:53:54.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:53:54.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:53:54.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:53:54.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:53:54.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:53:54.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:53:54.508 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:23.586+0000 7f92e9db1d80 -1 Falling back to public interface 2020-08-24T00:53:54.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:53:54.508 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:53:24.506+0000 7f92e9db1d80 -1 osd.2 29 log_to_monitors {default=true} 2020-08-24T00:53:54.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:53:54.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:53:54.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:53:54.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:53:54.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:53:54.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:53:54.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:53:54.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:53:54.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-08-24T00:53:54.510 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already in. 2020-08-24T00:53:54.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-08-24T00:53:54.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-08-24T00:53:54.511 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already in. 2020-08-24T00:53:54.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:53:54.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-08-24T00:53:54.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:53:54.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:53:54.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:53:54.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:53:54.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:53:54.512 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:53:54.513 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:53:54.513 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:53:54.513 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:53:54.513 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:53:54.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:53:54.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:53:54.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:53:54.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:53:54.514 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:53:54.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:53:54.515 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:53:54.515 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:53:54.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:53:54.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.516 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:53:54.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-08-24T00:53:54.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 150323855363 2020-08-24T00:53:54.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-150323855363' 2020-08-24T00:53:54.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:53:54.518 INFO:tasks.workunit.client.0.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-08-24T00:53:54.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-08-24T00:53:54.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z '' 2020-08-24T00:53:54.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: continue 2020-08-24T00:53:54.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:53:54.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:53:54.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-08-24T00:53:54.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 154618822659 2020-08-24T00:53:54.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-150323855363 2-154618822659' 2020-08-24T00:53:54.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-150323855363 2020-08-24T00:53:54.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:53:54.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-150323855363 2020-08-24T00:53:54.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=150323855363 2020-08-24T00:53:54.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2020-08-24T00:53:54.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:53:54.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 0 -lt 150323855363 2020-08-24T00:53:54.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:53:54.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:53:54.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:53:54.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 150323855364 -lt 150323855363 2020-08-24T00:53:54.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:53:54.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-154618822659 2020-08-24T00:53:54.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:53:54.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:53:54.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-154618822659 2020-08-24T00:53:54.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:53:54.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=154618822659 2020-08-24T00:53:54.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-08-24T00:53:54.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:53:54.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 154618822659 -lt 154618822659 2020-08-24T00:53:54.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:53:54.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T00:53:54.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T00:53:54.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T00:53:54.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T00:53:54.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:54.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.530 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.530 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.532 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.532 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.532 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:53:54.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:53:54.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:53:54.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:53:54.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:53:54.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:53:54.533 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:53:54.534 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:53:54.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:53:54.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:53:54.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-08-24T00:53:54.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-08-24T00:53:54.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:54.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.536 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.536 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.538 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.538 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.538 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:53:54.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:53:54.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:53:54.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:53:54.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:53:54.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:53:54.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:53:54.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:53:54.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:53:54.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:53:54.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-08-24T00:53:54.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-08-24T00:53:54.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:54.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.541 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.542 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.542 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.542 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.542 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.543 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.543 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.543 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.543 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:53:54.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:53:54.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:53:54.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:53:54.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:53:54.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:53:54.545 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:53:54.545 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:53:54.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:53:54.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:53:54.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-08-24T00:53:54.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-08-24T00:53:54.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:54.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.550 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:53:54.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:53:54.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:53:54.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:53:54.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:53:54.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:53:54.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:53:54.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:53:54.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:53:54.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:53:54.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-08-24T00:53:54.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-08-24T00:53:54.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:54.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.553 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.553 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.553 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:53:54.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:53:54.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:53:54.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:53:54.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:53:54.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:53:54.557 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:53:54.558 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:53:54.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:53:54.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:53:54.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-08-24T00:53:54.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-08-24T00:53:54.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:54.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.559 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.559 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:53:54.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:53:54.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:53:54.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:53:54.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:53:54.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:53:54.564 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:53:54.564 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:53:54.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:53:54.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:53:54.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-08-24T00:53:54.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-08-24T00:53:54.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:54.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:54.565 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:54.566 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:54.566 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:54.566 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:54.566 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:54.566 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:54.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:54.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:54.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:54.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:53:54.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:53:54.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:53:54.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:53:54.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:53:54.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:53:54.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:53:54.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:53:54.569 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:53:54.569 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:53:54.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:53:54.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:53:54.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-08-24T00:53:54.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:53:54.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-08-24T00:53:54.570 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836489 2020-08-24T00:53:54.570 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672968 2020-08-24T00:53:54.571 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509446 2020-08-24T00:53:54.571 INFO:tasks.workunit.client.0.smithi060.stdout:primary and soon to be divergent is 1 2020-08-24T00:53:54.571 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:53:54.571 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T00:52:33.848913+0000 0'0 28:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:53:54.571 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:53:54.572 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:53:54.572 INFO:tasks.workunit.client.0.smithi060.stdout:writing initial objects 2020-08-24T00:53:54.572 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836495 2020-08-24T00:53:54.572 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672974 2020-08-24T00:53:54.572 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509453 2020-08-24T00:53:54.573 INFO:tasks.workunit.client.0.smithi060.stdout:blackholing osds 0 2 2020-08-24T00:53:54.573 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:53:54.573 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-08-24T00:52:33.848913+0000 29'100 29:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:53:54.573 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:53:54.573 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:53:54.574 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:53:54.574 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T00:53:54.574 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:53:54.574 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:53:54.574 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T00:53:54.574 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:53:54.575 INFO:tasks.workunit.client.0.smithi060.stdout:writing divergent object 2020-08-24T00:53:54.575 INFO:tasks.workunit.client.0.smithi060.stdout:create missing divergent object 2020-08-24T00:53:54.575 INFO:tasks.workunit.client.0.smithi060.stdout:killing all the osds 2020-08-24T00:53:54.575 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:53:54.575 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 active+clean 2020-08-24T00:53:07.659821+0000 29'101 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:53:54.576 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:53:54.576 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:53:54.576 INFO:tasks.workunit.client.0.smithi060.stdout:bringing up non_divergent 0 2 2020-08-24T00:53:54.576 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:53:54.576 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 stale+active+clean 2020-08-24T00:53:07.659821+0000 29'101 29:122 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:53:54.577 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:53:54.577 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:53:54.577 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T00:53:54.577 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:53:54.577 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:53:54.578 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/16963,v1:127.0.0.1:6803/16963] [v2:127.0.0.1:6804/16963,v1:127.0.0.1:6805/16963] exists,up 0013942e-a74c-45d5-a0d9-ab468c0875f7 2020-08-24T00:53:54.578 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T00:53:54.578 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:53:54.578 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:53:54.578 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/17157,v1:127.0.0.1:6811/17157] [v2:127.0.0.1:6812/17157,v1:127.0.0.1:6813/17157] exists,up 2a35f1c2-02a9-48c5-b86c-a57c332d7bc5 2020-08-24T00:53:54.579 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 150323855363 2020-08-24T00:53:54.579 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 154618822659 2020-08-24T00:53:59.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:53:59.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:53:59.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:53:59.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:53:59.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:53:59.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:53:59.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:53:59.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:53:59.956 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:53:59.956 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:53:59.957 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:53:59.957 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:54:00.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:54:00.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:54:00.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:54:00.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:54:00.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:54:00.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:54:00.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:54:00.420 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:54:00.420 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:54:00.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:54:00.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:54:00.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-08-24T00:54:00.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-08-24T00:54:01.331 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:54:01.331 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_fingerprint": "623d4a85-28da-4983-bb9c-656221710cd4", 2020-08-24T00:54:01.332 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "15.2.4-661-g0a941e5", 2020-08-24T00:54:01.332 INFO:tasks.workunit.client.0.smithi060.stdout: "commit": "0a941e55043ccee6d35a8717bec55ff7581e67d8", 2020-08-24T00:54:01.332 INFO:tasks.workunit.client.0.smithi060.stdout: "timestamp": "2020-08-24T00:54:01.327317+0000", 2020-08-24T00:54:01.332 INFO:tasks.workunit.client.0.smithi060.stdout: "tag": "", 2020-08-24T00:54:01.332 INFO:tasks.workunit.client.0.smithi060.stdout: "health": { 2020-08-24T00:54:01.333 INFO:tasks.workunit.client.0.smithi060.stdout: "status": "HEALTH_WARN", 2020-08-24T00:54:01.333 INFO:tasks.workunit.client.0.smithi060.stdout: "checks": { 2020-08-24T00:54:01.333 INFO:tasks.workunit.client.0.smithi060.stdout: "OSDMAP_FLAGS": { 2020-08-24T00:54:01.333 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:54:01.333 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:54:01.334 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "nodown,noout,noin flag(s) set", 2020-08-24T00:54:01.334 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 17 2020-08-24T00:54:01.334 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.334 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [], 2020-08-24T00:54:01.334 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:54:01.335 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.335 INFO:tasks.workunit.client.0.smithi060.stdout: "OSD_DOWN": { 2020-08-24T00:54:01.335 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:54:01.336 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:54:01.336 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 osds down", 2020-08-24T00:54:01.336 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:54:01.337 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.337 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:54:01.337 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.337 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "osd.1 (root=default,host=smithi060) is down" 2020-08-24T00:54:01.337 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.338 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.338 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:54:01.338 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.339 INFO:tasks.workunit.client.0.smithi060.stdout: "PG_DEGRADED": { 2020-08-24T00:54:01.339 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:54:01.339 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:54:01.339 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-08-24T00:54:01.339 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:54:01.340 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.340 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:54:01.340 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.340 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:54:01.340 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.341 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.341 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:54:01.341 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.341 INFO:tasks.workunit.client.0.smithi060.stdout: "POOL_APP_NOT_ENABLED": { 2020-08-24T00:54:01.341 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:54:01.341 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:54:01.342 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 pool(s) do not have an application enabled", 2020-08-24T00:54:01.342 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:54:01.342 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.342 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:54:01.342 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.343 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "application not enabled on pool 'test'" 2020-08-24T00:54:01.343 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.343 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.343 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-08-24T00:54:01.343 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.346 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.346 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:54:01.346 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.347 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.347 INFO:tasks.workunit.client.0.smithi060.stdout: "mutes": [] 2020-08-24T00:54:01.347 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.347 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_first_committed": 1, 2020-08-24T00:54:01.347 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_last_committed": 1, 2020-08-24T00:54:01.348 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap": { 2020-08-24T00:54:01.348 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:54:01.348 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:54:01.348 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:52:09.971994Z", 2020-08-24T00:54:01.348 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:52:09.971994Z", 2020-08-24T00:54:01.349 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release": 15, 2020-08-24T00:54:01.349 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release_name": "octopus", 2020-08-24T00:54:01.349 INFO:tasks.workunit.client.0.smithi060.stdout: "features": { 2020-08-24T00:54:01.349 INFO:tasks.workunit.client.0.smithi060.stdout: "persistent": [ 2020-08-24T00:54:01.349 INFO:tasks.workunit.client.0.smithi060.stdout: "kraken", 2020-08-24T00:54:01.358 INFO:tasks.workunit.client.0.smithi060.stdout: "luminous", 2020-08-24T00:54:01.358 INFO:tasks.workunit.client.0.smithi060.stdout: "mimic", 2020-08-24T00:54:01.359 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap-prune", 2020-08-24T00:54:01.359 INFO:tasks.workunit.client.0.smithi060.stdout: "nautilus", 2020-08-24T00:54:01.359 INFO:tasks.workunit.client.0.smithi060.stdout: "octopus" 2020-08-24T00:54:01.359 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.360 INFO:tasks.workunit.client.0.smithi060.stdout: "optional": [] 2020-08-24T00:54:01.360 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.360 INFO:tasks.workunit.client.0.smithi060.stdout: "mons": [ 2020-08-24T00:54:01.360 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.360 INFO:tasks.workunit.client.0.smithi060.stdout: "rank": 0, 2020-08-24T00:54:01.361 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "a", 2020-08-24T00:54:01.361 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:54:01.361 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.361 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.361 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.362 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115", 2020-08-24T00:54:01.362 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 0 2020-08-24T00:54:01.362 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.362 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.362 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.363 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115/0", 2020-08-24T00:54:01.363 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:7115/0", 2020-08-24T00:54:01.363 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 0, 2020-08-24T00:54:01.363 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 0 2020-08-24T00:54:01.363 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.363 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.364 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.364 INFO:tasks.workunit.client.0.smithi060.stdout: "quorum": [ 2020-08-24T00:54:01.364 INFO:tasks.workunit.client.0.smithi060.stdout: 0 2020-08-24T00:54:01.364 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.364 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap": { 2020-08-24T00:54:01.365 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 39, 2020-08-24T00:54:01.365 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:54:01.365 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:52:10.631651+0000", 2020-08-24T00:54:01.365 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:53:26.393960+0000", 2020-08-24T00:54:01.365 INFO:tasks.workunit.client.0.smithi060.stdout: "last_up_change": "2020-08-24T00:53:24.598919+0000", 2020-08-24T00:54:01.374 INFO:tasks.workunit.client.0.smithi060.stdout: "last_in_change": "2020-08-24T00:53:26.109208+0000", 2020-08-24T00:54:01.374 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-08-24T00:54:01.374 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_num": 5800384, 2020-08-24T00:54:01.374 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_set": [ 2020-08-24T00:54:01.375 INFO:tasks.workunit.client.0.smithi060.stdout: "nodown", 2020-08-24T00:54:01.375 INFO:tasks.workunit.client.0.smithi060.stdout: "noin", 2020-08-24T00:54:01.375 INFO:tasks.workunit.client.0.smithi060.stdout: "noout", 2020-08-24T00:54:01.375 INFO:tasks.workunit.client.0.smithi060.stdout: "pglog_hardlimit", 2020-08-24T00:54:01.376 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snapdirs", 2020-08-24T00:54:01.376 INFO:tasks.workunit.client.0.smithi060.stdout: "recovery_deletes", 2020-08-24T00:54:01.376 INFO:tasks.workunit.client.0.smithi060.stdout: "sortbitwise" 2020-08-24T00:54:01.376 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.376 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_version": 7, 2020-08-24T00:54:01.377 INFO:tasks.workunit.client.0.smithi060.stdout: "full_ratio": 0.99000000953674316, 2020-08-24T00:54:01.377 INFO:tasks.workunit.client.0.smithi060.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-08-24T00:54:01.377 INFO:tasks.workunit.client.0.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2020-08-24T00:54:01.377 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_snapshot": "", 2020-08-24T00:54:01.377 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_max": 1, 2020-08-24T00:54:01.378 INFO:tasks.workunit.client.0.smithi060.stdout: "max_osd": 3, 2020-08-24T00:54:01.378 INFO:tasks.workunit.client.0.smithi060.stdout: "require_min_compat_client": "jewel", 2020-08-24T00:54:01.378 INFO:tasks.workunit.client.0.smithi060.stdout: "min_compat_client": "jewel", 2020-08-24T00:54:01.378 INFO:tasks.workunit.client.0.smithi060.stdout: "require_osd_release": "octopus", 2020-08-24T00:54:01.378 INFO:tasks.workunit.client.0.smithi060.stdout: "pools": [ 2020-08-24T00:54:01.379 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.379 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:54:01.379 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_name": "test", 2020-08-24T00:54:01.379 INFO:tasks.workunit.client.0.smithi060.stdout: "create_time": "2020-08-24T00:52:32.868696+0000", 2020-08-24T00:54:01.379 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": 1, 2020-08-24T00:54:01.379 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_names": "hashpspool", 2020-08-24T00:54:01.380 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:54:01.380 INFO:tasks.workunit.client.0.smithi060.stdout: "size": 3, 2020-08-24T00:54:01.380 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 2, 2020-08-24T00:54:01.380 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_rule": 0, 2020-08-24T00:54:01.380 INFO:tasks.workunit.client.0.smithi060.stdout: "object_hash": 2, 2020-08-24T00:54:01.381 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_autoscale_mode": "off", 2020-08-24T00:54:01.381 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num": 1, 2020-08-24T00:54:01.381 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num": 1, 2020-08-24T00:54:01.381 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num_target": 1, 2020-08-24T00:54:01.381 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_target": 1, 2020-08-24T00:54:01.385 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_pending": 1, 2020-08-24T00:54:01.386 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pg_merge_meta": { 2020-08-24T00:54:01.386 INFO:tasks.workunit.client.0.smithi060.stdout: "source_pgid": "0.0", 2020-08-24T00:54:01.386 INFO:tasks.workunit.client.0.smithi060.stdout: "ready_epoch": 0, 2020-08-24T00:54:01.386 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_started": 0, 2020-08-24T00:54:01.387 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": 0, 2020-08-24T00:54:01.387 INFO:tasks.workunit.client.0.smithi060.stdout: "source_version": "0'0", 2020-08-24T00:54:01.387 INFO:tasks.workunit.client.0.smithi060.stdout: "target_version": "0'0" 2020-08-24T00:54:01.387 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.387 INFO:tasks.workunit.client.0.smithi060.stdout: "last_change": "29", 2020-08-24T00:54:01.388 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend": "0", 2020-08-24T00:54:01.388 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2020-08-24T00:54:01.388 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2020-08-24T00:54:01.388 INFO:tasks.workunit.client.0.smithi060.stdout: "auid": 0, 2020-08-24T00:54:01.388 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_mode": "selfmanaged", 2020-08-24T00:54:01.388 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_seq": 0, 2020-08-24T00:54:01.389 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_epoch": 0, 2020-08-24T00:54:01.389 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_snaps": [], 2020-08-24T00:54:01.389 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps": "[]", 2020-08-24T00:54:01.389 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_bytes": 0, 2020-08-24T00:54:01.389 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_objects": 0, 2020-08-24T00:54:01.393 INFO:tasks.workunit.client.0.smithi060.stdout: "tiers": [], 2020-08-24T00:54:01.394 INFO:tasks.workunit.client.0.smithi060.stdout: "tier_of": -1, 2020-08-24T00:54:01.394 INFO:tasks.workunit.client.0.smithi060.stdout: "read_tier": -1, 2020-08-24T00:54:01.394 INFO:tasks.workunit.client.0.smithi060.stdout: "write_tier": -1, 2020-08-24T00:54:01.394 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_mode": "none", 2020-08-24T00:54:01.394 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_bytes": 0, 2020-08-24T00:54:01.395 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_objects": 0, 2020-08-24T00:54:01.395 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-08-24T00:54:01.395 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-08-24T00:54:01.395 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2020-08-24T00:54:01.395 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_flush_age": 0, 2020-08-24T00:54:01.396 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_evict_age": 0, 2020-08-24T00:54:01.396 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profile": "", 2020-08-24T00:54:01.396 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_params": { 2020-08-24T00:54:01.396 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "none" 2020-08-24T00:54:01.396 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.396 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_period": 0, 2020-08-24T00:54:01.397 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_count": 0, 2020-08-24T00:54:01.397 INFO:tasks.workunit.client.0.smithi060.stdout: "use_gmt_hitset": true, 2020-08-24T00:54:01.397 INFO:tasks.workunit.client.0.smithi060.stdout: "min_read_recency_for_promote": 0, 2020-08-24T00:54:01.397 INFO:tasks.workunit.client.0.smithi060.stdout: "min_write_recency_for_promote": 0, 2020-08-24T00:54:01.397 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2020-08-24T00:54:01.398 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_search_last_n": 0, 2020-08-24T00:54:01.398 INFO:tasks.workunit.client.0.smithi060.stdout: "grade_table": [], 2020-08-24T00:54:01.399 INFO:tasks.workunit.client.0.smithi060.stdout: "stripe_width": 0, 2020-08-24T00:54:01.399 INFO:tasks.workunit.client.0.smithi060.stdout: "expected_num_objects": 0, 2020-08-24T00:54:01.399 INFO:tasks.workunit.client.0.smithi060.stdout: "fast_read": false, 2020-08-24T00:54:01.400 INFO:tasks.workunit.client.0.smithi060.stdout: "options": {}, 2020-08-24T00:54:01.400 INFO:tasks.workunit.client.0.smithi060.stdout: "application_metadata": {} 2020-08-24T00:54:01.400 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.401 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.401 INFO:tasks.workunit.client.0.smithi060.stdout: "osds": [ 2020-08-24T00:54:01.401 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.401 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:54:01.402 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "0013942e-a74c-45d5-a0d9-ab468c0875f7", 2020-08-24T00:54:01.408 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:54:01.408 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:54:01.408 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:54:01.408 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:54:01.409 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 5, 2020-08-24T00:54:01.409 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 29, 2020-08-24T00:54:01.409 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 35, 2020-08-24T00:54:01.409 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 38, 2020-08-24T00:54:01.410 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 30, 2020-08-24T00:54:01.410 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:54:01.410 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:54:01.410 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.411 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.411 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.411 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6802", 2020-08-24T00:54:01.412 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.412 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.412 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.412 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.412 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6803", 2020-08-24T00:54:01.413 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.413 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.413 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.413 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.414 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:54:01.414 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.414 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.414 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.415 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6804", 2020-08-24T00:54:01.415 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.415 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.415 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.416 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.416 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6805", 2020-08-24T00:54:01.416 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.416 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.416 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.416 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.417 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:54:01.417 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.417 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.417 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.417 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6808", 2020-08-24T00:54:01.418 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.418 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.418 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.419 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.419 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6809", 2020-08-24T00:54:01.419 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.419 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.420 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.420 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.420 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:54:01.420 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.421 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.421 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.421 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6806", 2020-08-24T00:54:01.421 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.421 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.422 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.422 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.422 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6807", 2020-08-24T00:54:01.422 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 16963 2020-08-24T00:54:01.422 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.423 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.423 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.423 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6803/16963", 2020-08-24T00:54:01.423 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6805/16963", 2020-08-24T00:54:01.423 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6809/16963", 2020-08-24T00:54:01.424 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6807/16963", 2020-08-24T00:54:01.424 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:54:01.424 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:54:01.424 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:54:01.424 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.425 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.425 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.425 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:54:01.425 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "ed163f29-105f-4f32-a324-7646df6f5a35", 2020-08-24T00:54:01.425 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 0, 2020-08-24T00:54:01.425 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:54:01.426 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:54:01.426 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:54:01.426 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 0, 2020-08-24T00:54:01.426 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 0, 2020-08-24T00:54:01.426 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 10, 2020-08-24T00:54:01.427 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 23, 2020-08-24T00:54:01.427 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 31, 2020-08-24T00:54:01.427 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:54:01.427 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:54:01.434 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.434 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.434 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.434 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:54:01.435 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.435 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.435 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.435 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.435 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:54:01.436 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.436 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.436 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.436 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.436 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:54:01.436 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.437 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.437 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.437 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:54:01.437 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.437 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.438 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.438 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.438 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:54:01.438 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.438 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.439 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.439 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.439 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:54:01.439 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.439 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.439 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.440 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:54:01.440 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.440 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.440 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.440 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.441 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:54:01.441 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.441 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.441 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.441 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.442 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:54:01.442 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.442 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.442 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.442 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:54:01.443 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.443 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.443 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.443 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.443 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:54:01.444 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 12603 2020-08-24T00:54:01.444 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.444 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.444 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.444 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/12603", 2020-08-24T00:54:01.445 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/12603", 2020-08-24T00:54:01.445 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/12603", 2020-08-24T00:54:01.445 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/12603", 2020-08-24T00:54:01.445 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:54:01.445 INFO:tasks.workunit.client.0.smithi060.stdout: "exists" 2020-08-24T00:54:01.445 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.446 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.446 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.446 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:54:01.446 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "2a35f1c2-02a9-48c5-b86c-a57c332d7bc5", 2020-08-24T00:54:01.446 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:54:01.447 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:54:01.447 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:54:01.447 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:54:01.447 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 15, 2020-08-24T00:54:01.447 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 29, 2020-08-24T00:54:01.448 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 36, 2020-08-24T00:54:01.448 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 0, 2020-08-24T00:54:01.448 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 32, 2020-08-24T00:54:01.448 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:54:01.448 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:54:01.449 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.449 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.449 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.449 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:54:01.449 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.449 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.450 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.450 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.450 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:54:01.450 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.450 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.451 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.451 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.451 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:54:01.451 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.451 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.451 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.452 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:54:01.452 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.452 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.452 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.452 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.453 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:54:01.453 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.453 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.453 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.453 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.454 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:54:01.454 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.454 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.454 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.454 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:54:01.454 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.455 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.455 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.455 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.455 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:54:01.455 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.456 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.456 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.456 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.456 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:54:01.456 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:54:01.457 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.457 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:54:01.457 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:54:01.457 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.457 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.458 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.458 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:54:01.458 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:54:01.458 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 17157 2020-08-24T00:54:01.459 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.459 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.459 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.459 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/17157", 2020-08-24T00:54:01.459 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/17157", 2020-08-24T00:54:01.460 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/17157", 2020-08-24T00:54:01.460 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/17157", 2020-08-24T00:54:01.460 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:54:01.461 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:54:01.461 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:54:01.461 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.461 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.461 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.461 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_xinfo": [ 2020-08-24T00:54:01.462 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.462 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:54:01.462 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:53:18.167764+0000", 2020-08-24T00:54:01.462 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:54:01.462 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:54:01.463 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:54:01.463 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:54:01.463 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:52:17.566331+0000", 2020-08-24T00:54:01.463 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:54:01.463 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.464 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.464 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:54:01.464 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:53:18.685497+0000", 2020-08-24T00:54:01.464 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:54:01.464 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:54:01.465 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:54:01.465 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:54:01.465 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:52:23.925362+0000", 2020-08-24T00:54:01.465 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:54:01.465 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.465 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.466 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:54:01.466 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:53:19.202187+0000", 2020-08-24T00:54:01.466 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:54:01.466 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:54:01.466 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:54:01.467 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:54:01.467 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:52:30.461214+0000", 2020-08-24T00:54:01.467 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:54:01.467 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.467 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap": [], 2020-08-24T00:54:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap_items": [], 2020-08-24T00:54:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_temp": [], 2020-08-24T00:54:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_temp": [], 2020-08-24T00:54:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "blacklist": {}, 2020-08-24T00:54:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profiles": { 2020-08-24T00:54:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: "default": { 2020-08-24T00:54:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T00:54:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "2", 2020-08-24T00:54:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T00:54:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van" 2020-08-24T00:54:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps_queue": [], 2020-08-24T00:54:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: "new_removed_snaps": [], 2020-08-24T00:54:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: "new_purged_snaps": [], 2020-08-24T00:54:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_node_flags": {}, 2020-08-24T00:54:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: "device_class_flags": {} 2020-08-24T00:54:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_metadata": [ 2020-08-24T00:54:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.472 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:54:01.472 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:54:01.472 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6804/16963,v1:127.0.0.1:6805/16963]", 2020-08-24T00:54:01.472 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:54:01.472 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:54:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:54:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:54:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:54:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:54:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:54:01.474 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:54:01.474 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:54:01.474 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-08-24T00:54:01.475 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:54:01.475 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:54:01.476 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:54:01.476 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:54:01.476 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:54:01.476 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:54:01.476 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:54:01.477 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:54:01.477 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:54:01.477 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:54:01.477 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:54:01.477 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:54:01.478 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:54:01.478 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:54:01.478 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:54:01.478 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6802/16963,v1:127.0.0.1:6803/16963]", 2020-08-24T00:54:01.478 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:54:01.479 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/16963,v1:127.0.0.1:6809/16963]", 2020-08-24T00:54:01.479 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/16963,v1:127.0.0.1:6807/16963]", 2020-08-24T00:54:01.479 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:54:01.479 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:54:01.479 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:54:01.480 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:54:01.480 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:54:01.480 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:54:01.480 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:54:01.480 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:54:01.480 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:54:01.481 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/0", 2020-08-24T00:54:01.481 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:54:01.481 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:54:01.481 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:54:01.481 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.482 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.482 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:54:01.482 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:54:01.482 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/12603,v1:127.0.0.1:6813/12603]", 2020-08-24T00:54:01.482 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:54:01.483 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:54:01.483 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:54:01.483 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:54:01.483 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:54:01.483 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:54:01.484 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:54:01.484 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:54:01.484 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:54:01.484 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-08-24T00:54:01.484 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:54:01.485 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:54:01.485 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:54:01.485 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:54:01.485 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:54:01.486 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:54:01.486 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:54:01.486 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:54:01.486 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:54:01.487 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:54:01.487 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:54:01.488 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:54:01.488 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:54:01.488 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:54:01.488 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:54:01.488 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/12603,v1:127.0.0.1:6811/12603]", 2020-08-24T00:54:01.489 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:54:01.489 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/12603,v1:127.0.0.1:6817/12603]", 2020-08-24T00:54:01.489 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/12603,v1:127.0.0.1:6815/12603]", 2020-08-24T00:54:01.489 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:54:01.489 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:54:01.490 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:54:01.490 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:54:01.490 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:54:01.490 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:54:01.490 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:54:01.491 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:54:01.491 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:54:01.491 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/1", 2020-08-24T00:54:01.491 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:54:01.491 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:54:01.492 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:54:01.492 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.492 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.492 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:54:01.492 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:54:01.493 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/17157,v1:127.0.0.1:6813/17157]", 2020-08-24T00:54:01.493 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:54:01.493 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:54:01.493 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:54:01.493 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:54:01.493 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:54:01.494 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:54:01.494 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:54:01.494 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:54:01.494 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:54:01.494 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-08-24T00:54:01.495 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:54:01.495 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:54:01.495 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:54:01.495 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:54:01.495 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:54:01.496 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:54:01.496 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:54:01.496 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:54:01.496 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:54:01.496 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:54:01.497 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:54:01.497 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:54:01.497 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:54:01.497 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:54:01.498 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:54:01.498 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/17157,v1:127.0.0.1:6811/17157]", 2020-08-24T00:54:01.498 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:54:01.498 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/17157,v1:127.0.0.1:6817/17157]", 2020-08-24T00:54:01.498 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/17157,v1:127.0.0.1:6815/17157]", 2020-08-24T00:54:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:54:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:54:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:54:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:54:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:54:01.499 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:54:01.500 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:54:01.500 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:54:01.500 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:54:01.500 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/2", 2020-08-24T00:54:01.500 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:54:01.501 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:54:01.501 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:54:01.501 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.501 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.501 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_clean_epochs": { 2020-08-24T00:54:01.502 INFO:tasks.workunit.client.0.smithi060.stdout: "min_last_epoch_clean": 0, 2020-08-24T00:54:01.502 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": { 2020-08-24T00:54:01.502 INFO:tasks.workunit.client.0.smithi060.stdout: "per_pool": [] 2020-08-24T00:54:01.502 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.503 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_epochs": [ 2020-08-24T00:54:01.503 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.503 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:54:01.503 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 35 2020-08-24T00:54:01.503 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.503 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.504 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:54:01.504 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 10 2020-08-24T00:54:01.504 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.504 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.505 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:54:01.505 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 36 2020-08-24T00:54:01.506 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.507 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.507 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.507 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_first_committed": 1, 2020-08-24T00:54:01.507 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_last_committed": 39, 2020-08-24T00:54:01.508 INFO:tasks.workunit.client.0.smithi060.stdout: "crushmap": { 2020-08-24T00:54:01.508 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": [ 2020-08-24T00:54:01.508 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.508 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:54:01.508 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.0", 2020-08-24T00:54:01.508 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:54:01.509 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.509 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.510 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:54:01.510 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.1", 2020-08-24T00:54:01.510 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:54:01.510 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.510 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.511 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:54:01.511 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.2", 2020-08-24T00:54:01.511 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:54:01.511 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.511 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.511 INFO:tasks.workunit.client.0.smithi060.stdout: "types": [ 2020-08-24T00:54:01.512 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.512 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 0, 2020-08-24T00:54:01.512 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd" 2020-08-24T00:54:01.512 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.513 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.513 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:54:01.513 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "host" 2020-08-24T00:54:01.513 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.513 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.514 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 2, 2020-08-24T00:54:01.514 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "chassis" 2020-08-24T00:54:01.514 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.515 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.515 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 3, 2020-08-24T00:54:01.515 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "rack" 2020-08-24T00:54:01.517 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.518 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.518 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 4, 2020-08-24T00:54:01.518 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "row" 2020-08-24T00:54:01.518 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.519 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.519 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 5, 2020-08-24T00:54:01.519 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pdu" 2020-08-24T00:54:01.519 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.519 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.520 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 6, 2020-08-24T00:54:01.520 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pod" 2020-08-24T00:54:01.520 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.520 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.521 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 7, 2020-08-24T00:54:01.521 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "room" 2020-08-24T00:54:01.521 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.521 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.521 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 8, 2020-08-24T00:54:01.522 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "datacenter" 2020-08-24T00:54:01.522 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.522 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.522 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 9, 2020-08-24T00:54:01.523 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "zone" 2020-08-24T00:54:01.523 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.523 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.523 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 10, 2020-08-24T00:54:01.523 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "region" 2020-08-24T00:54:01.524 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.524 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.524 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:54:01.525 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "root" 2020-08-24T00:54:01.526 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.527 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.527 INFO:tasks.workunit.client.0.smithi060.stdout: "buckets": [ 2020-08-24T00:54:01.527 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.528 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -1, 2020-08-24T00:54:01.528 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default", 2020-08-24T00:54:01.528 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:54:01.528 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:54:01.528 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:54:01.529 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:54:01.529 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:54:01.529 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:54:01.529 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.530 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:54:01.530 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:54:01.530 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:54:01.530 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.531 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.531 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.531 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.531 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -2, 2020-08-24T00:54:01.532 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default~hdd", 2020-08-24T00:54:01.532 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:54:01.532 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:54:01.532 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:54:01.532 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:54:01.533 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:54:01.533 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:54:01.533 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.536 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:54:01.536 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:54:01.537 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:54:01.537 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.537 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.538 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.538 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.539 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:54:01.539 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060", 2020-08-24T00:54:01.539 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:54:01.540 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:54:01.540 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:54:01.540 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:54:01.540 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:54:01.541 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:54:01.541 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.542 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:54:01.542 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:54:01.543 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:54:01.543 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.543 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.544 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:54:01.544 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:54:01.544 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:54:01.545 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.545 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.545 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:54:01.546 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:54:01.546 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:54:01.546 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.547 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.547 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.547 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.548 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:54:01.548 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060~hdd", 2020-08-24T00:54:01.548 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:54:01.549 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:54:01.549 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:54:01.550 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:54:01.550 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:54:01.550 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:54:01.551 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.551 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:54:01.551 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:54:01.552 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:54:01.552 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.552 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.553 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:54:01.553 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:54:01.553 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:54:01.555 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.555 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.555 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:54:01.555 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:54:01.555 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:54:01.556 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.556 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.557 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.558 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.558 INFO:tasks.workunit.client.0.smithi060.stdout: "rules": [ 2020-08-24T00:54:01.558 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.558 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 0, 2020-08-24T00:54:01.559 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "replicated_rule", 2020-08-24T00:54:01.559 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 0, 2020-08-24T00:54:01.560 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:54:01.560 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 1, 2020-08-24T00:54:01.561 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 10, 2020-08-24T00:54:01.561 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T00:54:01.561 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.562 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T00:54:01.562 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T00:54:01.562 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T00:54:01.562 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.563 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.563 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_firstn", 2020-08-24T00:54:01.563 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T00:54:01.565 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T00:54:01.565 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.566 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.566 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T00:54:01.566 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.566 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:54:01.566 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.567 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.567 INFO:tasks.workunit.client.0.smithi060.stdout: "tunables": { 2020-08-24T00:54:01.569 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_tries": 0, 2020-08-24T00:54:01.569 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_fallback_tries": 0, 2020-08-24T00:54:01.570 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_total_tries": 50, 2020-08-24T00:54:01.570 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_descend_once": 1, 2020-08-24T00:54:01.574 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_vary_r": 1, 2020-08-24T00:54:01.575 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_stable": 1, 2020-08-24T00:54:01.575 INFO:tasks.workunit.client.0.smithi060.stdout: "straw_calc_version": 1, 2020-08-24T00:54:01.576 INFO:tasks.workunit.client.0.smithi060.stdout: "allowed_bucket_algs": 54, 2020-08-24T00:54:01.576 INFO:tasks.workunit.client.0.smithi060.stdout: "profile": "jewel", 2020-08-24T00:54:01.576 INFO:tasks.workunit.client.0.smithi060.stdout: "optimal_tunables": 1, 2020-08-24T00:54:01.577 INFO:tasks.workunit.client.0.smithi060.stdout: "legacy_tunables": 0, 2020-08-24T00:54:01.578 INFO:tasks.workunit.client.0.smithi060.stdout: "minimum_required_version": "jewel", 2020-08-24T00:54:01.578 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables": 1, 2020-08-24T00:54:01.578 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables2": 1, 2020-08-24T00:54:01.578 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v2_rules": 0, 2020-08-24T00:54:01.579 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables3": 1, 2020-08-24T00:54:01.579 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v3_rules": 0, 2020-08-24T00:54:01.579 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v4_buckets": 1, 2020-08-24T00:54:01.579 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables5": 1, 2020-08-24T00:54:01.579 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v5_rules": 0 2020-08-24T00:54:01.580 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.581 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_args": {} 2020-08-24T00:54:01.581 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.581 INFO:tasks.workunit.client.0.smithi060.stdout: "fsmap": { 2020-08-24T00:54:01.581 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:54:01.582 INFO:tasks.workunit.client.0.smithi060.stdout: "default_fscid": -1, 2020-08-24T00:54:01.582 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": { 2020-08-24T00:54:01.582 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": {}, 2020-08-24T00:54:01.582 INFO:tasks.workunit.client.0.smithi060.stdout: "ro_compat": {}, 2020-08-24T00:54:01.582 INFO:tasks.workunit.client.0.smithi060.stdout: "incompat": { 2020-08-24T00:54:01.583 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_1": "base v0.20", 2020-08-24T00:54:01.583 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_2": "client writeable ranges", 2020-08-24T00:54:01.583 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_3": "default file layouts on dirs", 2020-08-24T00:54:01.583 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_4": "dir inode in separate object", 2020-08-24T00:54:01.583 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_5": "mds uses versioned encoding", 2020-08-24T00:54:01.584 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_6": "dirfrag is stored in omap", 2020-08-24T00:54:01.584 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_8": "no anchor table", 2020-08-24T00:54:01.584 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_9": "file layout v2", 2020-08-24T00:54:01.584 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_10": "snaprealm v2" 2020-08-24T00:54:01.584 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.585 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.586 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_flags": { 2020-08-24T00:54:01.586 INFO:tasks.workunit.client.0.smithi060.stdout: "enable_multiple": false, 2020-08-24T00:54:01.586 INFO:tasks.workunit.client.0.smithi060.stdout: "ever_enabled_multiple": false 2020-08-24T00:54:01.586 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.587 INFO:tasks.workunit.client.0.smithi060.stdout: "standbys": [], 2020-08-24T00:54:01.587 INFO:tasks.workunit.client.0.smithi060.stdout: "filesystems": [] 2020-08-24T00:54:01.587 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.587 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_first_committed": 1, 2020-08-24T00:54:01.588 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_last_committed": 1, 2020-08-24T00:54:01.588 INFO:tasks.workunit.client.0.smithi060.stdout: "auth": { 2020-08-24T00:54:01.588 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:54:01.588 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 5, 2020-08-24T00:54:01.589 INFO:tasks.workunit.client.0.smithi060.stdout: "num_secrets": 3 2020-08-24T00:54:01.589 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.589 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T00:54:01.589 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_active": 1, 2020-08-24T00:54:01.589 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_unknown": 0, 2020-08-24T00:54:01.590 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osd": 3, 2020-08-24T00:54:01.590 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_sum": { 2020-08-24T00:54:01.590 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:54:01.590 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61700, 2020-08-24T00:54:01.590 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:54:01.590 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:54:01.591 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:54:01.591 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:54:01.591 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:54:01.591 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:54:01.591 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:54:01.592 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:54:01.592 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:54:01.592 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:54:01.592 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:54:01.592 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:54:01.593 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 100, 2020-08-24T00:54:01.593 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 100, 2020-08-24T00:54:01.593 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:54:01.593 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:54:01.593 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:54:01.594 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:54:01.594 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:54:01.594 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:54:01.594 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:54:01.594 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:54:01.594 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:54:01.595 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:54:01.595 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:54:01.595 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:54:01.595 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:54:01.595 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:54:01.596 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:54:01.596 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:54:01.598 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:54:01.598 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:54:01.598 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:54:01.599 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:54:01.599 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:54:01.599 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:54:01.599 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:54:01.599 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:54:01.600 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:54:01.600 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.600 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:54:01.600 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:54:01.600 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:54:01.600 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:54:01.601 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 0, 2020-08-24T00:54:01.601 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 0, 2020-08-24T00:54:01.601 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:54:01.601 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:54:01.601 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:54:01.602 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:54:01.602 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:54:01.602 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.602 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 16, 2020-08-24T00:54:01.602 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 16, 2020-08-24T00:54:01.603 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 2, 2020-08-24T00:54:01.603 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 2, 2020-08-24T00:54:01.603 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 0 2020-08-24T00:54:01.603 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.603 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum": { 2020-08-24T00:54:01.604 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:54:01.604 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:54:01.605 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T00:54:01.605 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:54:01.605 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:54:01.606 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:54:01.613 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:54:01.613 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3179264, 2020-08-24T00:54:01.614 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 33280, 2020-08-24T00:54:01.614 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 13, 2020-08-24T00:54:01.614 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145714, 2020-08-24T00:54:01.614 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311393536, 2020-08-24T00:54:01.615 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:54:01.615 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:54:01.616 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318866980864, 2020-08-24T00:54:01.616 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:54:01.616 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 34078720, 2020-08-24T00:54:01.616 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 445445, 2020-08-24T00:54:01.616 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:54:01.617 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:54:01.617 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:54:01.617 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 13498, 2020-08-24T00:54:01.617 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221211974 2020-08-24T00:54:01.617 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.618 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:54:01.618 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:54:01.618 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:54:01.618 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 1, 2020-08-24T00:54:01.618 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:54:01.619 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:54:01.619 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:54:01.619 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.619 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:54:01.619 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:54:01.619 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:54:01.620 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:54:01.620 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:54:01.620 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.620 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:54:01.620 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:54:01.621 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.621 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum_by_class": { 2020-08-24T00:54:01.621 INFO:tasks.workunit.client.0.smithi060.stdout: "hdd": { 2020-08-24T00:54:01.621 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:54:01.621 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:54:01.622 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T00:54:01.622 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:54:01.622 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:54:01.622 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:54:01.623 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:54:01.623 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3179264, 2020-08-24T00:54:01.623 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 33280, 2020-08-24T00:54:01.624 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 13, 2020-08-24T00:54:01.624 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145714, 2020-08-24T00:54:01.624 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311393536, 2020-08-24T00:54:01.624 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:54:01.624 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:54:01.625 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318866980864, 2020-08-24T00:54:01.625 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:54:01.625 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 34078720, 2020-08-24T00:54:01.627 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 445445, 2020-08-24T00:54:01.627 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:54:01.627 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:54:01.627 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:54:01.627 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 13498, 2020-08-24T00:54:01.628 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221211974 2020-08-24T00:54:01.628 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.628 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:54:01.628 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:54:01.628 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:54:01.629 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 1, 2020-08-24T00:54:01.629 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:54:01.629 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:54:01.629 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:54:01.629 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.630 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:54:01.630 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:54:01.630 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:54:01.630 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:54:01.630 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:54:01.631 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.631 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:54:01.631 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:54:01.631 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.631 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.631 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_stats": [ 2020-08-24T00:54:01.632 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.632 INFO:tasks.workunit.client.0.smithi060.stdout: "poolid": 1, 2020-08-24T00:54:01.632 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T00:54:01.632 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:54:01.632 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61700, 2020-08-24T00:54:01.633 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:54:01.633 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:54:01.633 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:54:01.633 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:54:01.633 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:54:01.634 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:54:01.634 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:54:01.634 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:54:01.634 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:54:01.635 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:54:01.635 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:54:01.635 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:54:01.635 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 100, 2020-08-24T00:54:01.635 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 100, 2020-08-24T00:54:01.635 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:54:01.636 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:54:01.636 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:54:01.636 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:54:01.636 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:54:01.636 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:54:01.637 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:54:01.637 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:54:01.637 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:54:01.637 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:54:01.637 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:54:01.638 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:54:01.638 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:54:01.638 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:54:01.638 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:54:01.638 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:54:01.638 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:54:01.639 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:54:01.639 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:54:01.639 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:54:01.639 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:54:01.639 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:54:01.640 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:54:01.640 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:54:01.640 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:54:01.640 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.640 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:54:01.640 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:54:01.641 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:54:01.641 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:54:01.641 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 19660800, 2020-08-24T00:54:01.641 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 185100, 2020-08-24T00:54:01.641 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:54:01.642 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:54:01.642 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:54:01.642 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:54:01.642 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:54:01.642 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.643 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 16, 2020-08-24T00:54:01.643 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 16, 2020-08-24T00:54:01.643 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 2, 2020-08-24T00:54:01.643 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 2, 2020-08-24T00:54:01.643 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 3 2020-08-24T00:54:01.643 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.644 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.644 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_stats": [ 2020-08-24T00:54:01.644 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.644 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:54:01.644 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 150323855370 2020-08-24T00:54:01.645 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.645 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.645 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:54:01.645 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 42949672977 2020-08-24T00:54:01.646 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.646 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.646 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:54:01.646 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 154618822666 2020-08-24T00:54:01.646 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.647 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.647 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_state": [ 2020-08-24T00:54:01.647 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.647 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "active+undersized+degraded", 2020-08-24T00:54:01.648 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 1 2020-08-24T00:54:01.649 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.649 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.649 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_osd": [ 2020-08-24T00:54:01.649 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.650 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:54:01.650 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:54:01.650 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T00:54:01.650 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:54:01.651 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.651 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.651 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:54:01.651 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:54:01.652 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 0, 2020-08-24T00:54:01.652 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:54:01.652 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.652 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.653 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:54:01.653 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 1, 2020-08-24T00:54:01.653 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T00:54:01.653 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:54:01.654 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.654 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.654 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": [ 2020-08-24T00:54:01.654 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:54:01.655 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:54:01.655 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": {} 2020-08-24T00:54:01.655 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.655 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:54:01.656 INFO:tasks.workunit.client.0.smithi060.stdout: "servicemap": { 2020-08-24T00:54:01.656 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:54:01.656 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:52:10.632572+0000", 2020-08-24T00:54:01.656 INFO:tasks.workunit.client.0.smithi060.stdout: "services": {} 2020-08-24T00:54:01.657 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:54:01.658 INFO:tasks.workunit.client.0.smithi060.stdout: "paxos": { 2020-08-24T00:54:01.658 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:54:01.659 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 346, 2020-08-24T00:54:01.659 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pn": 0, 2020-08-24T00:54:01.660 INFO:tasks.workunit.client.0.smithi060.stdout: "accepted_pn": 0 2020-08-24T00:54:01.661 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:54:01.661 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:54:01.664 INFO:tasks.workunit.client.0.smithi060.stderr:report 548095831 2020-08-24T00:54:01.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-08-24T00:54:01.665 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-08-24T00:54:01.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-08-24T00:54:01.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-08-24T00:54:01.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:01.666 INFO:tasks.workunit.client.0.smithi060.stdout:writing non-divergent object existing_4 2020-08-24T00:54:01.701 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:01.702 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-08-24T00:53:26.685582+0000 29'100 39:121 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:01.702 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:01.702 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:01.707 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:01.712 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-08-24T00:54:01.862 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery 2020-08-24T00:54:01.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-08-24T00:54:01.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:02.206 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:02.207 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-08-24T00:53:26.685582+0000 29'100 39:121 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:02.207 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:02.207 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:02.208 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:02.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-08-24T00:54:02.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-08-24T00:54:02.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T00:54:02.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T00:54:02.220 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:54:02.220 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:02.220 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:02.222 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T00:54:02.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-08-24T00:54:02.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok set_recovery_delay 100000 2020-08-24T00:54:02.313 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T00:54:02.321 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-08-24T00:54:02.322 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-08-24T00:54:02.323 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T00:54:02.323 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T00:54:02.323 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:54:02.324 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:02.324 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:02.324 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T00:54:02.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-08-24T00:54:02.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok set_recovery_delay 100000 2020-08-24T00:54:02.422 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T00:54:02.438 INFO:tasks.workunit.client.0.smithi060.stdout:revive divergent 1 2020-08-24T00:54:02.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-08-24T00:54:02.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:02.767 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:02.767 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-08-24T00:53:26.685582+0000 39'101 39:122 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:02.767 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:02.768 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:02.770 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:02.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-08-24T00:54:03.434 INFO:tasks.workunit.client.0.smithi060.stderr:noup is set 2020-08-24T00:54:03.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-08-24T00:54:03.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:54:03.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:54:03.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T00:54:03.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:54:03.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T00:54:03.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:54:03.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:54:03.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:54:03.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:54:03.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T00:54:03.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T00:54:03.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:54:03.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:54:03.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:54:03.460 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:54:03.461 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:54:03.461 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:54:03.461 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:54:03.463 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:03.463 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:03.464 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:03.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:03.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:54:03.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:54:03.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:54:03.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:54:03.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:54:03.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:03.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:54:03.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:54:03.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T00:54:03.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T00:54:03.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:03.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T00:54:03.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T00:54:03.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:54:03.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:54:03.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:54:03.470 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:54:03.490 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:03.486+0000 7f78317cbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:03.505 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:03.498+0000 7f78317cbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:03.505 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:03.502+0000 7f78317cbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:03.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-08-24T00:54:03.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-08-24T00:54:04.041 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:04.038+0000 7f78317cbd80 -1 Falling back to public interface 2020-08-24T00:54:05.000 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:04.994+0000 7f78317cbd80 -1 osd.1 29 log_to_monitors {default=true} 2020-08-24T00:54:08.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-08-24T00:54:08.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:08.837 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery divergent 2020-08-24T00:54:09.186 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:09.186 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-08-24T00:53:26.685582+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:09.187 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:09.187 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:09.188 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:09.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-08-24T00:54:09.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T00:54:09.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T00:54:09.198 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:54:09.199 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:09.199 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:09.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T00:54:09.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-08-24T00:54:09.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.1.asok set_recovery_delay 100000 2020-08-24T00:54:09.295 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T00:54:09.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-08-24T00:54:09.922 INFO:tasks.workunit.client.0.smithi060.stderr:noup is unset 2020-08-24T00:54:09.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-08-24T00:54:09.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:54:09.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:54:09.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:54:09.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:54:09.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:54:09.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:54:09.933 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:54:09.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:54:09.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:54:10.293 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 35 up_thru 38 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/16963,v1:127.0.0.1:6803/16963] [v2:127.0.0.1:6804/16963,v1:127.0.0.1:6805/16963] exists,up 0013942e-a74c-45d5-a0d9-ab468c0875f7 2020-08-24T00:54:10.293 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:54:10.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:54:10.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:54:10.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:54:10.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-08-24T00:54:10.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:54:10.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:54:10.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:54:10.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:54:10.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:54:10.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:54:10.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:54:10.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:54:10.648 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/18585,v1:127.0.0.1:6819/18585] [v2:127.0.0.1:6820/18585,v1:127.0.0.1:6821/18585] exists,up ed163f29-105f-4f32-a324-7646df6f5a35 2020-08-24T00:54:10.648 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:54:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:54:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:54:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:54:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-08-24T00:54:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:54:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:54:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:54:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:54:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:54:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:54:10.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:54:10.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:54:11.034 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [15,29) [v2:127.0.0.1:6810/17157,v1:127.0.0.1:6811/17157] [v2:127.0.0.1:6812/17157,v1:127.0.0.1:6813/17157] exists,up 2a35f1c2-02a9-48c5-b86c-a57c332d7bc5 2020-08-24T00:54:11.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:54:11.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:54:11.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:54:11.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:11.400 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:11.400 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2020-08-24T00:54:09.980200+0000 29'101 43:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:11.400 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:11.401 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:11.402 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:11.417 INFO:tasks.workunit.client.0.smithi060.stdout:wait for peering 2020-08-24T00:54:11.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-08-24T00:54:11.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:11.749 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:11.750 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 17 17 peering 2020-08-24T00:54:09.980200+0000 29'101 43:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:11.750 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:11.750 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:11.751 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:11.759 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-08-24T00:54:22.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-08-24T00:54:22.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:22.825 INFO:tasks.workunit.client.0.smithi060.stdout:killing divergent 1 2020-08-24T00:54:23.184 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:23.184 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 2 2 0 0 61854 0 0 17 17 active+recovering+degraded+wait 2020-08-24T00:54:10.459336+0000 39'101 44:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:23.184 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:23.184 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:23.187 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:23.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-08-24T00:54:23.195 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:54:23.196 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:54:23.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:54:23.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:54:23.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:54:23.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:54:23.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-08-24T00:54:23.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:23.307 INFO:tasks.workunit.client.0.smithi060.stdout:reviving divergent 1 2020-08-24T00:54:23.642 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:23.643 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 2 2 0 0 61854 0 0 17 17 active+recovering+degraded+wait 2020-08-24T00:54:10.459336+0000 39'101 44:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:23.643 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:23.643 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:23.645 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:23.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-08-24T00:54:23.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:54:23.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:54:23.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T00:54:23.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:54:23.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T00:54:23.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:54:23.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:54:23.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:54:23.657 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:54:23.657 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T00:54:23.657 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T00:54:23.657 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:54:23.657 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:54:23.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:54:23.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:54:23.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:54:23.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:54:23.660 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:54:23.660 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:23.661 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:23.661 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:23.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:23.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:54:23.662 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:54:23.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:54:23.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:54:23.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:54:23.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:23.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:54:23.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:54:23.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T00:54:23.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T00:54:23.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:23.666 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T00:54:23.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T00:54:23.666 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:54:23.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:54:23.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:54:23.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:54:23.699 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:23.685+0000 7f87bb608d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:23.699 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:23.689+0000 7f87bb608d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:23.699 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:23.689+0000 7f87bb608d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:24.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-08-24T00:54:24.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:54:24.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:54:24.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:54:24.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:54:24.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:54:24.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:54:24.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:54:24.029 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:54:24.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:54:24.225 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:24.221+0000 7f87bb608d80 -1 Falling back to public interface 2020-08-24T00:54:24.404 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 31 last_clean_interval [10,29) [v2:127.0.0.1:6818/18585,v1:127.0.0.1:6819/18585] [v2:127.0.0.1:6820/18585,v1:127.0.0.1:6821/18585] exists,up ed163f29-105f-4f32-a324-7646df6f5a35 2020-08-24T00:54:24.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:54:24.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:54:24.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:54:24.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-08-24T00:54:25.141 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:25.137+0000 7f87bb608d80 -1 osd.1 44 log_to_monitors {default=true} 2020-08-24T00:54:44.407 INFO:tasks.workunit.client.0.smithi060.stdout:allowing recovery 2020-08-24T00:54:58.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-08-24T00:54:58.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:58.831 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:58.839 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 0 0 0 62471 0 0 18 18 active+clean 2020-08-24T00:54:26.867387+0000 44'102 48:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:58.849 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:58.849 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:58.849 INFO:tasks.workunit.client.0.smithi060.stdout:reading divergent objects 2020-08-24T00:54:58.850 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:54:58.850 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 0 0 0 62471 0 0 18 18 active+clean 2020-08-24T00:54:26.867387+0000 44'102 48:152 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:52:32.919701+0000 0'0 2020-08-24T00:52:32.919701+0000 0 2020-08-24T00:54:58.850 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:54:58.850 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:54:58.851 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-08-24T00:53:06.776449+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-08-24T00:54:58.851 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 20 _merge_object_divergent_entries: keeping 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-08-24T00:53:06.776449+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-08-24T00:54:58.851 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T00:54:58.852 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 29'100 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-08-24T00:54:58.852 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-08-24T00:54:58.853 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-08-24T00:54:58.853 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 29'101 2020-08-24T00:54:58.854 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 29'101 (29'100) modify 1:c156033e:::existing_100:head by client.4526.0:1 2020-08-24T00:53:06.776449+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-08-24T00:54:58.854 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:54:10.330+0000 7f7810408700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-08-24T00:54:58.855 INFO:tasks.workunit.client.0.smithi060.stdout:success 2020-08-24T00:54:58.856 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 0df00c36-03db-4e00-963a-1bcde275259c 2020-08-24T00:54:58.856 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:54:58.856 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T00:54:58.856 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T00:54:58.856 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:54:58.857 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:54:58.857 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/19874,v1:127.0.0.1:6803/19874] [v2:127.0.0.1:6804/19874,v1:127.0.0.1:6805/19874] exists,up 0df00c36-03db-4e00-963a-1bcde275259c 2020-08-24T00:54:58.857 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 838fa102-f839-43c9-9f7e-b2b276ac878e 2020-08-24T00:54:58.858 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:54:58.858 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T00:54:58.859 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:58.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-08-24T00:54:58.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-08-24T00:54:58.860 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T00:54:58.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-08-24T00:54:58.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-08-24T00:54:58.861 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T00:54:58.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-08-24T00:54:58.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-08-24T00:54:58.862 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T00:54:58.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-08-24T00:54:58.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-08-24T00:54:58.863 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:54:58.863 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-08-24T00:54:58.864 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-08-24T00:54:58.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:54:58.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-08-24T00:54:58.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:54:58.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-08-24T00:54:58.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:54:58.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-08-24T00:54:58.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:54:58.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-08-24T00:54:58.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-08-24T00:54:58.867 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-08-24T00:54:58.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T00:54:58.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-08-24T00:54:58.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T00:54:58.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-08-24T00:54:58.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-08-24T00:54:58.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-08-24T00:54:58.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-08-24T00:54:58.870 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' does not exist 2020-08-24T00:54:58.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-08-24T00:54:58.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:54:58.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:54:58.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:54:58.872 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:54:58.872 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:54:58.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:54:58.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-08-24T00:54:58.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T00:54:58.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T00:54:58.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T00:54:58.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:54:58.875 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:54:58.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:54:58.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:54:58.876 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:54:58.876 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:54:58.876 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:54:58.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:54:58.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:54:58.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:54:58.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:54:58.878 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:54:58.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:54:58.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:54:58.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:54:58.880 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:54:58.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:54:58.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T00:54:58.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T00:54:58.881 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:54:58.881 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.882 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T00:54:58.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:54:58.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:54:58.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-08-24T00:54:58.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-08-24T00:54:58.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-08-24T00:54:58.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-08-24T00:54:58.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T00:54:58.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T00:54:58.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T00:54:58.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:54:58.886 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:54:58.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:54:58.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:54:58.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:54:58.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:54:58.888 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:54:58.888 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:54:58.888 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:54:58.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:54:58.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:54:58.890 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:54:58.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:54:58.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:54:58.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:54:58.891 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:54:58.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:54:58.892 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T00:54:58.892 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T00:54:58.892 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:54:58.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T00:54:58.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:54:58.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:54:58.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-08-24T00:54:58.895 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T00:54:58.895 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.895 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.11819 2020-08-24T00:54:58.896 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T00:54:58.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-08-24T00:54:58.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T00:54:58.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-08-24T00:54:58.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-08-24T00:54:58.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-08-24T00:54:58.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-08-24T00:54:58.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-08-24T00:54:58.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-08-24T00:54:58.899 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-08-24T00:54:58.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-08-24T00:54:58.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-08-24T00:54:58.900 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:54:58.900 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:54:58.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-08-24T00:54:58.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-08-24T00:54:58.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T00:54:58.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T00:54:58.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T00:54:58.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-08-24T00:54:58.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-08-24T00:54:58.903 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T00:54:58.904 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:54:58.904 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:54:58.904 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:54:58.905 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.905 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:58.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T00:54:58.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T00:54:58.906 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T00:54:58.907 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:54:58.907 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:54:58.907 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T00:54:58.908 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T00:54:58.908 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:54:58.908 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:54:58.909 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:54:58.909 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:54:58.909 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.910 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.910 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T00:54:58.910 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:54:58.911 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get fsid 2020-08-24T00:54:58.911 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T00:54:58.911 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:54:58.912 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:54:58.912 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T00:54:58.912 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:54:58.913 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:54:58.913 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:54:58.913 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T00:54:58.914 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:54:58.914 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.915 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.915 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T00:54:58.915 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:54:58.915 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get mon_host 2020-08-24T00:54:58.915 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-08-24T00:54:58.915 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-08-24T00:54:58.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T00:54:58.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T00:54:58.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T00:54:58.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-08-24T00:54:58.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T00:54:58.917 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T00:54:58.917 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:54:58.917 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:54:58.917 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:54:58.917 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.918 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.918 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:58.918 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:54:58.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:54:58.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-08-24T00:54:58.919 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-08-24T00:54:58.919 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:54:58.919 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:54:58.919 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T00:54:58.919 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:54:58.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-08-24T00:54:58.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:54:58.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:54:58.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:54:58.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:54:58.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T00:54:58.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T00:54:58.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:54:58.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:54:58.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:54:58.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:54:58.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:54:58.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:54:58.922 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:54:58.922 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.922 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.923 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:58.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:58.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:54:58.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:54:58.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:54:58.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:54:58.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:54:58.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:54:58.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:54:58.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:58.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:54:58.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:54:58.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-08-24T00:54:58.925 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:54:58.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0df00c36-03db-4e00-963a-1bcde275259c 2020-08-24T00:54:58.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 0df00c36-03db-4e00-963a-1bcde275259c' 2020-08-24T00:54:58.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:54:58.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDYD0Nf5GyeHhAAglY/CVOysf8UTDZcIO7h3Q== 2020-08-24T00:54:58.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDYD0Nf5GyeHhAAglY/CVOysf8UTDZcIO7h3Q=="}' 2020-08-24T00:54:58.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0df00c36-03db-4e00-963a-1bcde275259c -i td/divergent-priors/0/new.json 2020-08-24T00:54:58.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-08-24T00:54:58.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDYD0Nf5GyeHhAAglY/CVOysf8UTDZcIO7h3Q== --osd-uuid 0df00c36-03db-4e00-963a-1bcde275259c 2020-08-24T00:54:58.927 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:48.877+0000 7f80eb6b8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.927 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:48.881+0000 7f80eb6b8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.927 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:48.885+0000 7f80eb6b8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:48.885+0000 7f80eb6b8d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:54:58.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:48.885+0000 7f80eb6b8d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:54:58.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:48.885+0000 7f80eb6b8d80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:54:58.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:48.937+0000 7f80eb6b8d80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-08-24T00:54:58.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:49.321+0000 7f80eb6b8d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:54:58.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-08-24T00:54:58.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:54:58.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T00:54:58.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:54:58.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T00:54:58.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:58.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:54:58.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:54:58.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:54:58.930 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:52.545+0000 7f864e8a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.930 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:52.545+0000 7f864e8a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.931 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:52.549+0000 7f864e8a6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T00:54:58.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:54:58.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:54:58.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:54:58.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:54:58.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:54:58.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:54:58.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:54:58.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:54:58.933 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:53.089+0000 7f864e8a6d80 -1 Falling back to public interface 2020-08-24T00:54:58.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:54:58.933 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:53.893+0000 7f864e8a6d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T00:54:58.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:54:58.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:54:58.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:54:58.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:54:58.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:54:58.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:54:58.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:54:58.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:54:58.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-08-24T00:54:58.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-08-24T00:54:58.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:54:58.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:54:58.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T00:54:58.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:54:58.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-08-24T00:54:58.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:54:58.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:54:58.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:54:58.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:54:58.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T00:54:58.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T00:54:58.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:54:58.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:54:58.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:54:58.938 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:54:58.938 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:54:58.939 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:54:58.939 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:54:58.939 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:54:58.939 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:54:58.939 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:58.939 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:54:58.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:54:58.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:54:58.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:54:58.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:54:58.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:54:58.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:54:58.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:54:58.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:58.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:54:58.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:54:58.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-08-24T00:54:58.942 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:54:58.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=838fa102-f839-43c9-9f7e-b2b276ac878e 2020-08-24T00:54:58.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 838fa102-f839-43c9-9f7e-b2b276ac878e' 2020-08-24T00:54:58.942 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:54:58.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDeD0Nfjh5bKRAAgqs8mVk5/rKkn3vlP2tS+A== 2020-08-24T00:54:58.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDeD0Nfjh5bKRAAgqs8mVk5/rKkn3vlP2tS+A=="}' 2020-08-24T00:54:58.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 838fa102-f839-43c9-9f7e-b2b276ac878e -i td/divergent-priors/1/new.json 2020-08-24T00:54:58.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-08-24T00:54:58.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDeD0Nfjh5bKRAAgqs8mVk5/rKkn3vlP2tS+A== --osd-uuid 838fa102-f839-43c9-9f7e-b2b276ac878e 2020-08-24T00:54:58.944 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.349+0000 7fee82042d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.944 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.353+0000 7fee82042d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.944 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.353+0000 7fee82042d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:58.945 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.357+0000 7fee82042d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:54:58.945 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.357+0000 7fee82042d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:54:58.945 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.357+0000 7fee82042d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:54:58.945 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.425+0000 7fee82042d80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-08-24T00:54:58.945 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:55.841+0000 7fee82042d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:54:58.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-08-24T00:54:58.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:54:58.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T00:54:58.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:54:59.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T00:54:59.121 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:54:59.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:54:59.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:54:59.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:54:59.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:54:59.156 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:59.153+0000 7f494136fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:59.159 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:59.153+0000 7f494136fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:59.160 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:59.157+0000 7f494136fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:54:59.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T00:54:59.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:54:59.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:54:59.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:54:59.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:54:59.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:54:59.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:54:59.489 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:54:59.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:54:59.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:54:59.710 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:54:59.705+0000 7f494136fd80 -1 Falling back to public interface 2020-08-24T00:54:59.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:55:00.520 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:00.509+0000 7f494136fd80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T00:55:00.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:55:00.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:55:00.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:55:00.862 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:55:00.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:55:00.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:55:01.246 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/20229,v1:127.0.0.1:6811/20229] [v2:127.0.0.1:6812/20229,v1:127.0.0.1:6813/20229] exists,up 838fa102-f839-43c9-9f7e-b2b276ac878e 2020-08-24T00:55:01.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:55:01.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:55:01.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:55:01.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-08-24T00:55:01.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-08-24T00:55:01.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:55:01.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:55:01.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T00:55:01.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:55:01.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-08-24T00:55:01.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:55:01.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:55:01.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:55:01.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:55:01.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T00:55:01.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T00:55:01.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:55:01.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:55:01.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:55:01.251 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:55:01.251 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:55:01.251 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:55:01.253 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:55:01.253 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:55:01.253 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:55:01.254 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:55:01.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:55:01.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:55:01.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:55:01.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:55:01.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:55:01.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:55:01.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:55:01.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:55:01.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:55:01.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:55:01.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:55:01.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-08-24T00:55:01.256 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:55:01.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4423a225-da95-4821-9d96-64b9877e85a1 2020-08-24T00:55:01.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 4423a225-da95-4821-9d96-64b9877e85a1' 2020-08-24T00:55:01.257 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:55:01.257 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 4423a225-da95-4821-9d96-64b9877e85a1 2020-08-24T00:55:01.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDlD0Nfeeh+EBAAZeODUxU5F0LUOZaBD4SHpw== 2020-08-24T00:55:01.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDlD0Nfeeh+EBAAZeODUxU5F0LUOZaBD4SHpw=="}' 2020-08-24T00:55:01.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4423a225-da95-4821-9d96-64b9877e85a1 -i td/divergent-priors/2/new.json 2020-08-24T00:55:01.740 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T00:55:01.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-08-24T00:55:01.758 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDlD0Nfeeh+EBAAZeODUxU5F0LUOZaBD4SHpw== --osd-uuid 4423a225-da95-4821-9d96-64b9877e85a1 2020-08-24T00:55:01.793 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:01.789+0000 7f2ca9467d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:55:01.796 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:01.793+0000 7f2ca9467d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:55:01.797 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:01.793+0000 7f2ca9467d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:55:01.798 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:01.793+0000 7f2ca9467d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:55:01.798 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:01.793+0000 7f2ca9467d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:55:01.798 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:01.793+0000 7f2ca9467d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:55:01.868 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:01.865+0000 7f2ca9467d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-08-24T00:55:02.334 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:02.329+0000 7f2ca9467d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:55:05.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-08-24T00:55:05.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:55:05.003 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T00:55:05.003 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T00:55:05.003 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:55:05.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T00:55:05.470 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T00:55:05.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:55:05.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:55:05.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:55:05.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:55:05.515 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:05.501+0000 7f96ee36bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:55:05.515 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:05.505+0000 7f96ee36bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:55:05.515 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:05.505+0000 7f96ee36bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:55:05.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T00:55:05.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:55:05.808 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:55:05.808 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:55:05.808 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:55:05.808 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:55:05.808 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:55:05.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:55:05.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:55:05.809 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:55:06.046 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:06.041+0000 7f96ee36bd80 -1 Falling back to public interface 2020-08-24T00:55:06.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:55:06.811 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:06.805+0000 7f96ee36bd80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T00:55:07.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:55:07.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:55:07.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:55:07.194 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:55:07.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:55:07.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:55:07.610 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/20587,v1:127.0.0.1:6819/20587] [v2:127.0.0.1:6820/20587,v1:127.0.0.1:6821/20587] exists,up 4423a225-da95-4821-9d96-64b9877e85a1 2020-08-24T00:55:07.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:55:07.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:55:07.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:55:07.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-08-24T00:55:08.260 INFO:tasks.workunit.client.0.smithi060.stderr:noout is set 2020-08-24T00:55:08.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-08-24T00:55:08.980 INFO:tasks.workunit.client.0.smithi060.stderr:noin is set 2020-08-24T00:55:08.993 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-08-24T00:55:09.693 INFO:tasks.workunit.client.0.smithi060.stderr:nodown is set 2020-08-24T00:55:09.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-08-24T00:55:09.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-08-24T00:55:10.248 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T00:55:10.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T00:55:11.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-08-24T00:55:11.889 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 size to 3 2020-08-24T00:55:11.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-08-24T00:55:12.610 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 min_size to 2 2020-08-24T00:55:12.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-08-24T00:55:12.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:55:12.618 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:55:12.998 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:55:12.999 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:55:12.999 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:55:12.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:55:12.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:13.000 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:55:13.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-08-24T00:55:13.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836486 2020-08-24T00:55:13.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486' 2020-08-24T00:55:13.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:13.188 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:55:13.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-08-24T00:55:13.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672966 2020-08-24T00:55:13.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-08-24T00:55:13.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:13.355 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:55:13.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-08-24T00:55:13.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542148 2020-08-24T00:55:13.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-60129542148' 2020-08-24T00:55:13.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:13.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836486 2020-08-24T00:55:13.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:13.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:55:13.516 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2020-08-24T00:55:13.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:13.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836486 2020-08-24T00:55:13.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-08-24T00:55:13.519 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836486 2020-08-24T00:55:13.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:55:13.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836484 -lt 21474836486 2020-08-24T00:55:13.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:55:14.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:55:14.900 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:55:15.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836486 -lt 21474836486 2020-08-24T00:55:15.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:15.263 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672966 2020-08-24T00:55:15.263 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:15.265 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:55:15.265 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672966 2020-08-24T00:55:15.266 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:15.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672966 2020-08-24T00:55:15.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-08-24T00:55:15.268 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672966 2020-08-24T00:55:15.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:55:15.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672966 -lt 42949672966 2020-08-24T00:55:15.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:15.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542148 2020-08-24T00:55:15.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:15.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:55:15.661 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:15.661 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542148 2020-08-24T00:55:15.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542148 2020-08-24T00:55:15.662 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-08-24T00:55:15.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:55:15.662 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542148 2020-08-24T00:55:16.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542148 -lt 60129542148 2020-08-24T00:55:16.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-08-24T00:55:16.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:55:16.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:55:16.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:55:16.039 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:55:16.040 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T00:55:16.040 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:55:16.041 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:55:16.041 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:55:16.041 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:55:16.042 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:55:16.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:55:16.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:55:16.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:55:16.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:55:16.229 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:55:16.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:55:16.604 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:55:16.605 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:55:16.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:55:16.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:16.605 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:55:16.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-08-24T00:55:16.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836489 2020-08-24T00:55:16.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489' 2020-08-24T00:55:16.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:16.769 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:55:16.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-08-24T00:55:16.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672969 2020-08-24T00:55:16.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489 1-42949672969' 2020-08-24T00:55:16.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:16.936 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:55:17.102 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-08-24T00:55:17.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542150 2020-08-24T00:55:17.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489 1-42949672969 2-60129542150' 2020-08-24T00:55:17.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:17.104 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836489 2020-08-24T00:55:17.104 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:17.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:55:17.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2020-08-24T00:55:17.113 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:17.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836489 2020-08-24T00:55:17.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-08-24T00:55:17.113 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:55:17.114 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836489 2020-08-24T00:55:17.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836487 -lt 21474836489 2020-08-24T00:55:17.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:55:18.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:55:18.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:55:18.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836489 -lt 21474836489 2020-08-24T00:55:18.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:18.865 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672969 2020-08-24T00:55:18.865 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:18.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:55:18.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672969 2020-08-24T00:55:18.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:18.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672969 2020-08-24T00:55:18.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-08-24T00:55:18.870 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672969 2020-08-24T00:55:18.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:55:19.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672969 -lt 42949672969 2020-08-24T00:55:20.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:20.215 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542150 2020-08-24T00:55:20.215 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:20.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:55:20.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542150 2020-08-24T00:55:20.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:20.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542150 2020-08-24T00:55:20.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-08-24T00:55:20.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:55:20.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542150 -lt 60129542150 2020-08-24T00:55:20.217 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:55:20.217 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:55:20.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:55:20.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:55:20.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:55:20.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:55:20.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:55:20.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:55:20.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:55:20.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:55:20.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:55:20.220 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542150 2020-08-24T00:55:20.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T00:55:20.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:55:20.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:55:20.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:55:20.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T00:55:20.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T00:55:20.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T00:55:20.910 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-08-24T00:55:20.911 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-08-24T00:55:21.256 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:55:21.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-08-24T00:55:21.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-08-24T00:55:21.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:55:21.267 INFO:tasks.workunit.client.0.smithi060.stdout:primary and soon to be divergent is 1 2020-08-24T00:55:21.610 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:55:21.610 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T00:55:11.063897+0000 0'0 26:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:55:21.611 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:55:21.611 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:55:21.613 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:55:21.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-08-24T00:55:21.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-08-24T00:55:21.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-08-24T00:55:21.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-08-24T00:55:21.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-08-24T00:55:21.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-08-24T00:55:21.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-08-24T00:55:21.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-08-24T00:55:21.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-08-24T00:55:21.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-08-24T00:55:21.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-08-24T00:55:21.624 INFO:tasks.workunit.client.0.smithi060.stdout:writing initial objects 2020-08-24T00:55:21.625 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-08-24T00:55:21.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:21.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-08-24T00:55:21.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:21.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-08-24T00:55:21.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:21.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-08-24T00:55:22.099 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:22.099 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-08-24T00:55:22.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:22.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-08-24T00:55:22.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:22.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-08-24T00:55:22.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:22.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-08-24T00:55:22.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:22.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-08-24T00:55:23.012 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:23.012 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-08-24T00:55:23.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:23.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-08-24T00:55:23.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:23.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-08-24T00:55:23.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:23.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-08-24T00:55:23.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:23.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-08-24T00:55:23.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:23.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-08-24T00:55:23.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:23.905 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-08-24T00:55:24.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:24.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-08-24T00:55:24.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:24.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-08-24T00:55:24.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:24.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-08-24T00:55:24.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:24.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-08-24T00:55:24.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:24.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-08-24T00:55:24.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:24.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-08-24T00:55:24.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-08-24T00:55:25.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:25.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-08-24T00:55:25.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:25.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-08-24T00:55:25.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:25.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-08-24T00:55:25.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:25.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-08-24T00:55:25.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:25.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-08-24T00:55:25.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:25.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-08-24T00:55:25.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:25.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-08-24T00:55:26.145 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:26.145 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-08-24T00:55:26.287 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:26.287 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-08-24T00:55:26.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:26.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-08-24T00:55:26.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:26.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-08-24T00:55:26.761 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:26.761 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-08-24T00:55:26.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:26.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-08-24T00:55:27.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:27.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-08-24T00:55:27.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:27.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-08-24T00:55:27.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:27.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-08-24T00:55:27.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:27.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-08-24T00:55:27.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:27.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-08-24T00:55:27.892 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:27.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-08-24T00:55:28.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:28.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-08-24T00:55:28.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:28.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-08-24T00:55:28.328 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:28.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-08-24T00:55:28.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:28.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-08-24T00:55:28.633 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:28.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-08-24T00:55:28.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:28.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-08-24T00:55:28.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:28.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-08-24T00:55:29.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:29.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-08-24T00:55:29.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:29.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-08-24T00:55:29.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:29.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-08-24T00:55:29.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:29.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-08-24T00:55:29.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:29.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-08-24T00:55:29.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:29.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-08-24T00:55:29.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:29.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-08-24T00:55:30.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:30.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-08-24T00:55:30.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:30.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-08-24T00:55:30.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:30.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-08-24T00:55:30.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:30.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-08-24T00:55:30.743 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:30.743 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-08-24T00:55:30.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:30.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-08-24T00:55:31.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:31.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-08-24T00:55:31.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:31.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-08-24T00:55:31.332 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:31.333 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-08-24T00:55:31.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:31.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-08-24T00:55:31.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:31.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-08-24T00:55:31.772 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:31.773 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-08-24T00:55:31.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:31.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-08-24T00:55:32.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:32.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-08-24T00:55:32.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:32.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-08-24T00:55:32.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:32.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-08-24T00:55:32.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:32.638 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-08-24T00:55:32.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:32.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-08-24T00:55:32.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:32.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-08-24T00:55:33.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:33.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-08-24T00:55:33.284 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:33.285 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-08-24T00:55:33.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:33.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-08-24T00:55:33.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:33.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-08-24T00:55:33.732 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:33.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-08-24T00:55:33.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:33.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-08-24T00:55:34.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:34.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-08-24T00:55:34.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:34.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-08-24T00:55:34.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:34.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-08-24T00:55:34.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:34.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-08-24T00:55:34.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:34.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-08-24T00:55:34.832 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:34.833 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-08-24T00:55:35.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:35.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-08-24T00:55:35.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:35.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-08-24T00:55:35.281 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:35.282 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-08-24T00:55:35.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:35.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-08-24T00:55:35.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:35.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-08-24T00:55:35.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:35.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-08-24T00:55:35.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:35.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-08-24T00:55:36.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:36.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-08-24T00:55:36.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:36.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-08-24T00:55:36.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:36.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-08-24T00:55:36.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:36.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-08-24T00:55:36.692 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:36.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-08-24T00:55:36.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:36.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-08-24T00:55:37.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-08-24T00:55:37.006 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-08-24T00:55:37.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:55:37.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-08-24T00:55:37.172 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:55:37.172 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:55:37.172 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:55:37.172 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:55:37.172 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:55:37.173 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:55:37.173 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:55:37.174 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:55:37.174 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:55:37.174 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:55:37.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:55:37.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:55:37.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:55:37.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:55:37.312 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:55:37.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:55:37.680 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:55:37.681 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:55:37.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:55:37.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:37.682 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:55:37.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836495 2020-08-24T00:55:37.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836495 2020-08-24T00:55:37.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495' 2020-08-24T00:55:37.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:37.847 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:55:38.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672975 2020-08-24T00:55:38.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672975 2020-08-24T00:55:38.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672975' 2020-08-24T00:55:38.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:55:38.010 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:55:38.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542156 2020-08-24T00:55:38.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542156 2020-08-24T00:55:38.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672975 2-60129542156' 2020-08-24T00:55:38.172 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:38.172 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836495 2020-08-24T00:55:38.173 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:38.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:55:38.176 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2020-08-24T00:55:38.177 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:38.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836495 2020-08-24T00:55:38.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-08-24T00:55:38.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:55:38.179 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836495 2020-08-24T00:55:38.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836495 -lt 21474836495 2020-08-24T00:55:38.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:38.543 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672975 2020-08-24T00:55:38.544 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:38.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:55:38.545 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672975 2020-08-24T00:55:38.546 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:38.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672975 2020-08-24T00:55:38.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2020-08-24T00:55:38.548 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672975 2020-08-24T00:55:38.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:55:38.911 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672973 -lt 42949672975 2020-08-24T00:55:38.912 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:55:39.913 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:55:39.917 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:55:40.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672975 -lt 42949672975 2020-08-24T00:55:40.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:55:40.289 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542156 2020-08-24T00:55:40.289 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:55:40.291 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:55:40.292 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542156 2020-08-24T00:55:40.292 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:55:40.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542156 2020-08-24T00:55:40.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2020-08-24T00:55:40.293 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542156 2020-08-24T00:55:40.294 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:55:40.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542157 -lt 60129542156 2020-08-24T00:55:40.664 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:55:40.664 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:55:40.665 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:55:41.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:55:41.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:55:41.118 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:55:41.119 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:55:41.119 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:55:41.120 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:55:41.120 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:55:41.120 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:55:41.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T00:55:41.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:55:41.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:55:41.470 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:55:41.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T00:55:41.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T00:55:41.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T00:55:41.929 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-08-24T00:55:41.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-08-24T00:55:41.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-08-24T00:55:41.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-08-24T00:55:41.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-08-24T00:55:42.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-08-24T00:55:42.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-08-24T00:55:42.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:55:42.313 INFO:tasks.workunit.client.0.smithi060.stdout:blackholing osds 0 2 2020-08-24T00:55:42.673 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:55:42.673 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-08-24T00:55:11.063897+0000 27'100 27:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:55:42.674 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:55:42.674 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:55:42.674 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:55:42.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:55:42.684 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-08-24T00:55:42.684 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T00:55:42.684 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T00:55:42.685 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:55:42.688 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:55:42.688 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:55:42.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T00:55:42.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-08-24T00:55:42.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok config set objectstore_blackhole 1 2020-08-24T00:55:42.775 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:55:42.776 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T00:55:42.776 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:55:42.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:55:42.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-08-24T00:55:42.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T00:55:42.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T00:55:42.785 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:55:42.785 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:55:42.786 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:55:42.786 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T00:55:42.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-08-24T00:55:42.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok config set objectstore_blackhole 1 2020-08-24T00:55:42.875 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:55:42.875 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T00:55:42.875 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:55:42.882 INFO:tasks.workunit.client.0.smithi060.stdout:create new divergent objects 2020-08-24T00:55:42.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-08-24T00:55:42.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-08-24T00:55:42.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T00:55:42.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-08-24T00:55:42.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T00:55:42.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-08-24T00:55:42.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-08-24T00:55:42.886 INFO:tasks.workunit.client.0.smithi060.stdout:writing divergent objects 2020-08-24T00:55:42.887 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-08-24T00:55:42.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-08-24T00:55:42.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-08-24T00:55:42.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-08-24T00:55:42.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-08-24T00:55:42.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-08-24T00:55:42.891 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-08-24T00:55:42.891 INFO:tasks.workunit.client.0.smithi060.stdout:remove divergent objects 2020-08-24T00:55:42.892 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-08-24T00:55:42.892 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-08-24T00:55:42.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-08-24T00:55:42.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-08-24T00:55:42.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-08-24T00:55:42.894 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-08-24T00:55:42.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-08-24T00:55:42.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-08-24T00:55:42.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-08-24T00:55:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-08-24T00:56:30.394 INFO:tasks.workunit.client.0.smithi060.stderr:rados: no process found 2020-08-24T00:56:30.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-08-24T00:56:30.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:56:30.396 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:56:30.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-08-24T00:56:30.397 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:56:30.397 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:56:30.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:56:30.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:56:30.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:56:30.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:56:30.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-08-24T00:56:30.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-08-24T00:56:30.399 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already down. 2020-08-24T00:56:30.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-08-24T00:56:30.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-08-24T00:56:30.399 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already down. 2020-08-24T00:56:30.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-08-24T00:56:30.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-08-24T00:56:30.400 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already down. 2020-08-24T00:56:30.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:56:30.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-08-24T00:56:30.401 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already out. 2020-08-24T00:56:30.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:56:30.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-08-24T00:56:30.401 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already out. 2020-08-24T00:56:30.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-08-24T00:56:30.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:56:30.402 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:56:30.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:56:30.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-08-24T00:56:30.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:56:30.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:56:30.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-08-24T00:56:30.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:56:30.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-08-24T00:56:30.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:56:30.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:56:30.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:56:30.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:56:30.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T00:56:30.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T00:56:30.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:56:30.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:56:30.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:56:30.405 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:56:30.405 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:56:30.406 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:56:30.406 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:56:30.407 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:56:30.407 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:56:30.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:56:30.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:56:30.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:56:30.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:56:30.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:56:30.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:56:30.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:56:30.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:56:30.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:56:30.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:56:30.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-08-24T00:56:30.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-08-24T00:56:30.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:56:30.410 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-08-24T00:56:30.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-08-24T00:56:30.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:56:30.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:56:30.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:56:30.411 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:56.441+0000 7ff6628f7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:56:30.412 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:56.445+0000 7ff6628f7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:56:30.412 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:56.445+0000 7ff6628f7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:56:30.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-08-24T00:56:30.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:56:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:56:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:56:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:56:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:56:30.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:56:30.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:56:30.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:56:30.414 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:56.985+0000 7ff6628f7d80 -1 Falling back to public interface 2020-08-24T00:56:30.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:56:30.415 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:57.937+0000 7ff6628f7d80 -1 osd.0 27 log_to_monitors {default=true} 2020-08-24T00:56:30.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:56:30.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:56:30.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:56:30.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:56:30.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:56:30.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:56:30.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:56:30.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:56:30.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:56:30.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-08-24T00:56:30.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:56:30.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:56:30.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-08-24T00:56:30.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:56:30.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-08-24T00:56:30.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:56:30.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:56:30.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:56:30.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:56:30.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T00:56:30.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T00:56:30.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:56:30.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:56:30.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:56:30.421 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:56:30.421 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:56:30.421 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:56:30.421 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:56:30.421 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:56:30.422 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:56:30.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:56:30.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:56:30.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:56:30.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:56:30.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:56:30.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:56:30.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:56:30.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:56:30.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:56:30.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:56:30.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-08-24T00:56:30.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-08-24T00:56:30.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:56:30.424 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-08-24T00:56:30.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-08-24T00:56:30.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:56:30.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:56:30.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:56:30.425 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:58.569+0000 7fd850cb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:56:30.426 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:58.569+0000 7fd850cb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:56:30.426 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:58.573+0000 7fd850cb7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:56:30.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-08-24T00:56:30.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:56:30.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:56:30.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:56:30.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:56:30.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:56:30.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:56:30.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:56:30.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:56:30.428 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:55:59.117+0000 7fd850cb7d80 -1 Falling back to public interface 2020-08-24T00:56:30.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:56:30.429 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:56:00.077+0000 7fd850cb7d80 -1 osd.2 27 log_to_monitors {default=true} 2020-08-24T00:56:30.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:56:30.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:56:30.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:56:30.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:56:30.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:56:30.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:56:30.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:56:30.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:56:30.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:56:30.431 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-08-24T00:56:30.431 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already in. 2020-08-24T00:56:30.431 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:56:30.431 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-08-24T00:56:30.431 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already in. 2020-08-24T00:56:30.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:56:30.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-08-24T00:56:30.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:56:30.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:56:30.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:56:30.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:56:30.433 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:56:30.433 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:56:30.433 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:56:30.433 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:56:30.434 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:56:30.434 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:56:30.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:56:30.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:56:30.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:56:30.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:56:30.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:56:30.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:56:30.436 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:56:30.436 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:56:30.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:56:30.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:56:30.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:56:30.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-08-24T00:56:30.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 141733920771 2020-08-24T00:56:30.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-141733920771' 2020-08-24T00:56:30.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:56:30.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:56:30.438 INFO:tasks.workunit.client.0.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-08-24T00:56:30.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-08-24T00:56:30.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z '' 2020-08-24T00:56:30.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: continue 2020-08-24T00:56:30.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:56:30.439 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:56:30.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-08-24T00:56:30.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 146028888067 2020-08-24T00:56:30.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-141733920771 2-146028888067' 2020-08-24T00:56:30.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:56:30.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-141733920771 2020-08-24T00:56:30.441 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:56:30.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:56:30.441 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-141733920771 2020-08-24T00:56:30.441 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:56:30.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=141733920771 2020-08-24T00:56:30.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 141733920771' 2020-08-24T00:56:30.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:56:30.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 0 -lt 141733920771 2020-08-24T00:56:30.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:56:30.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:56:30.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:56:30.443 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 141733920772 -lt 141733920771 2020-08-24T00:56:30.443 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:56:30.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-146028888067 2020-08-24T00:56:30.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:56:30.444 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:56:30.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-146028888067 2020-08-24T00:56:30.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:56:30.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=146028888067 2020-08-24T00:56:30.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 146028888067' 2020-08-24T00:56:30.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:56:30.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 146028888067 -lt 146028888067 2020-08-24T00:56:30.446 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.446 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.446 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:56:30.447 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T00:56:30.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T00:56:30.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T00:56:30.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T00:56:30.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:30.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.451 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.451 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.451 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.453 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.453 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:30.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:30.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:30.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:30.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:30.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:30.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:30.455 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:30.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:30.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:30.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-08-24T00:56:30.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-08-24T00:56:30.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:30.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.459 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.459 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:30.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:30.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:30.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:30.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:30.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:30.461 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:30.461 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:30.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:30.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:30.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-08-24T00:56:30.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-08-24T00:56:30.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:30.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.464 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.464 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.464 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.464 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:30.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:30.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:30.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:30.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:30.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:30.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:30.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:30.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:30.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:30.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-08-24T00:56:30.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-08-24T00:56:30.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:30.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.468 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.468 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.468 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.470 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.470 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.471 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:30.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:30.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:30.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:30.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:30.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:30.472 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:30.472 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:30.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:30.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:30.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-08-24T00:56:30.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-08-24T00:56:30.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:30.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.476 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.476 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.476 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:30.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:30.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:30.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:30.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:30.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:30.478 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:30.479 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:30.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:30.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:30.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-08-24T00:56:30.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-08-24T00:56:30.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:30.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.483 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:30.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:30.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:30.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:30.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:30.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:30.484 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:30.484 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:30.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:30.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:30.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-08-24T00:56:30.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-08-24T00:56:30.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:30.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:30.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:30.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:30.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:30.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:30.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:30.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:30.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:30.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:30.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:30.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:30.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:30.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:30.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:30.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:30.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:30.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:30.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:30.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:30.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:30.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:30.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:30.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-08-24T00:56:30.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:30.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-08-24T00:56:30.491 INFO:tasks.workunit.client.0.smithi060.stdout:killing all the osds 2020-08-24T00:56:30.492 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:56:30.492 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 active+clean 2020-08-24T00:55:11.063897+0000 27'106 27:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:56:30.492 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:56:30.492 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:56:30.492 INFO:tasks.workunit.client.0.smithi060.stdout:bringing up non_divergent 0 2 2020-08-24T00:56:30.493 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:56:30.493 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 stale+active+clean 2020-08-24T00:55:11.063897+0000 27'106 27:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:56:30.493 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:56:30.493 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:56:30.493 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T00:56:30.494 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:56:30.494 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:56:30.494 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up out weight 0 up_from 33 up_thru 0 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/24614,v1:127.0.0.1:6803/24614] [v2:127.0.0.1:6804/24614,v1:127.0.0.1:6805/24614] exists,up 0df00c36-03db-4e00-963a-1bcde275259c 2020-08-24T00:56:30.494 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T00:56:30.494 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:56:30.495 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:56:30.495 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up out weight 0 up_from 34 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/24808,v1:127.0.0.1:6811/24808] [v2:127.0.0.1:6812/24808,v1:127.0.0.1:6813/24808] exists,up 4423a225-da95-4821-9d96-64b9877e85a1 2020-08-24T00:56:30.495 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 141733920771 2020-08-24T00:56:30.495 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 146028888067 2020-08-24T00:56:35.155 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:35.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:35.157 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:35.157 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:35.157 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:35.158 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:35.158 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:35.158 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:35.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:35.529 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:35.530 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:35.530 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:35.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:35.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:35.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:35.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:35.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:35.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:35.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:35.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:35.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:36.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:36.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:36.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-08-24T00:56:36.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-08-24T00:56:36.897 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:56:36.898 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_fingerprint": "832b028f-196d-40ef-b01c-c1eaa6abfa9d", 2020-08-24T00:56:36.898 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "15.2.4-661-g0a941e5", 2020-08-24T00:56:36.898 INFO:tasks.workunit.client.0.smithi060.stdout: "commit": "0a941e55043ccee6d35a8717bec55ff7581e67d8", 2020-08-24T00:56:36.899 INFO:tasks.workunit.client.0.smithi060.stdout: "timestamp": "2020-08-24T00:56:36.894478+0000", 2020-08-24T00:56:36.899 INFO:tasks.workunit.client.0.smithi060.stdout: "tag": "", 2020-08-24T00:56:36.899 INFO:tasks.workunit.client.0.smithi060.stdout: "health": { 2020-08-24T00:56:36.899 INFO:tasks.workunit.client.0.smithi060.stdout: "status": "HEALTH_WARN", 2020-08-24T00:56:36.899 INFO:tasks.workunit.client.0.smithi060.stdout: "checks": { 2020-08-24T00:56:36.900 INFO:tasks.workunit.client.0.smithi060.stdout: "OSDMAP_FLAGS": { 2020-08-24T00:56:36.900 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:56:36.900 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:56:36.900 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "nodown,noout,noin flag(s) set", 2020-08-24T00:56:36.900 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 17 2020-08-24T00:56:36.901 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.901 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [], 2020-08-24T00:56:36.901 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:56:36.901 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.901 INFO:tasks.workunit.client.0.smithi060.stdout: "OSD_DOWN": { 2020-08-24T00:56:36.901 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:56:36.902 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:56:36.902 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 osds down", 2020-08-24T00:56:36.902 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:56:36.902 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.902 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:56:36.903 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.903 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "osd.1 (root=default,host=smithi060) is down" 2020-08-24T00:56:36.903 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.903 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.903 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:56:36.904 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.904 INFO:tasks.workunit.client.0.smithi060.stdout: "PG_DEGRADED": { 2020-08-24T00:56:36.904 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:56:36.904 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:56:36.904 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-08-24T00:56:36.904 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:56:36.905 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.905 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:56:36.905 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.905 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:56:36.905 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.906 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.906 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:56:36.906 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.906 INFO:tasks.workunit.client.0.smithi060.stdout: "POOL_APP_NOT_ENABLED": { 2020-08-24T00:56:36.906 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:56:36.907 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:56:36.907 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 pool(s) do not have an application enabled", 2020-08-24T00:56:36.907 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:56:36.907 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.907 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:56:36.907 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.908 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "application not enabled on pool 'test'" 2020-08-24T00:56:36.908 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.908 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.908 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-08-24T00:56:36.908 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.909 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.909 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:56:36.909 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.909 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.909 INFO:tasks.workunit.client.0.smithi060.stdout: "mutes": [] 2020-08-24T00:56:36.910 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.910 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_first_committed": 1, 2020-08-24T00:56:36.910 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_last_committed": 1, 2020-08-24T00:56:36.910 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap": { 2020-08-24T00:56:36.910 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:56:36.910 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:56:36.911 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:54:46.900407Z", 2020-08-24T00:56:36.911 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:54:46.900407Z", 2020-08-24T00:56:36.911 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release": 15, 2020-08-24T00:56:36.911 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release_name": "octopus", 2020-08-24T00:56:36.911 INFO:tasks.workunit.client.0.smithi060.stdout: "features": { 2020-08-24T00:56:36.912 INFO:tasks.workunit.client.0.smithi060.stdout: "persistent": [ 2020-08-24T00:56:36.912 INFO:tasks.workunit.client.0.smithi060.stdout: "kraken", 2020-08-24T00:56:36.912 INFO:tasks.workunit.client.0.smithi060.stdout: "luminous", 2020-08-24T00:56:36.912 INFO:tasks.workunit.client.0.smithi060.stdout: "mimic", 2020-08-24T00:56:36.912 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap-prune", 2020-08-24T00:56:36.913 INFO:tasks.workunit.client.0.smithi060.stdout: "nautilus", 2020-08-24T00:56:36.913 INFO:tasks.workunit.client.0.smithi060.stdout: "octopus" 2020-08-24T00:56:36.913 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.913 INFO:tasks.workunit.client.0.smithi060.stdout: "optional": [] 2020-08-24T00:56:36.913 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.913 INFO:tasks.workunit.client.0.smithi060.stdout: "mons": [ 2020-08-24T00:56:36.914 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.914 INFO:tasks.workunit.client.0.smithi060.stdout: "rank": 0, 2020-08-24T00:56:36.914 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "a", 2020-08-24T00:56:36.914 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:56:36.914 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.915 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.915 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.915 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115", 2020-08-24T00:56:36.915 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 0 2020-08-24T00:56:36.915 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.916 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.916 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.916 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115/0", 2020-08-24T00:56:36.916 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:7115/0", 2020-08-24T00:56:36.916 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 0, 2020-08-24T00:56:36.916 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 0 2020-08-24T00:56:36.917 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.917 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.917 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.917 INFO:tasks.workunit.client.0.smithi060.stdout: "quorum": [ 2020-08-24T00:56:36.917 INFO:tasks.workunit.client.0.smithi060.stdout: 0 2020-08-24T00:56:36.918 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.918 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap": { 2020-08-24T00:56:36.918 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 37, 2020-08-24T00:56:36.918 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:56:36.918 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:54:47.614196+0000", 2020-08-24T00:56:36.918 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:56:01.914722+0000", 2020-08-24T00:56:36.919 INFO:tasks.workunit.client.0.smithi060.stdout: "last_up_change": "2020-08-24T00:56:00.174103+0000", 2020-08-24T00:56:36.919 INFO:tasks.workunit.client.0.smithi060.stdout: "last_in_change": "2020-08-24T00:56:01.605181+0000", 2020-08-24T00:56:36.919 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-08-24T00:56:36.919 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_num": 5800384, 2020-08-24T00:56:36.919 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_set": [ 2020-08-24T00:56:36.920 INFO:tasks.workunit.client.0.smithi060.stdout: "nodown", 2020-08-24T00:56:36.920 INFO:tasks.workunit.client.0.smithi060.stdout: "noin", 2020-08-24T00:56:36.920 INFO:tasks.workunit.client.0.smithi060.stdout: "noout", 2020-08-24T00:56:36.920 INFO:tasks.workunit.client.0.smithi060.stdout: "pglog_hardlimit", 2020-08-24T00:56:36.920 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snapdirs", 2020-08-24T00:56:36.921 INFO:tasks.workunit.client.0.smithi060.stdout: "recovery_deletes", 2020-08-24T00:56:36.921 INFO:tasks.workunit.client.0.smithi060.stdout: "sortbitwise" 2020-08-24T00:56:36.921 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.921 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_version": 7, 2020-08-24T00:56:36.921 INFO:tasks.workunit.client.0.smithi060.stdout: "full_ratio": 0.99000000953674316, 2020-08-24T00:56:36.921 INFO:tasks.workunit.client.0.smithi060.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-08-24T00:56:36.922 INFO:tasks.workunit.client.0.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2020-08-24T00:56:36.922 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_snapshot": "", 2020-08-24T00:56:36.922 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_max": 1, 2020-08-24T00:56:36.922 INFO:tasks.workunit.client.0.smithi060.stdout: "max_osd": 3, 2020-08-24T00:56:36.922 INFO:tasks.workunit.client.0.smithi060.stdout: "require_min_compat_client": "jewel", 2020-08-24T00:56:36.923 INFO:tasks.workunit.client.0.smithi060.stdout: "min_compat_client": "jewel", 2020-08-24T00:56:36.923 INFO:tasks.workunit.client.0.smithi060.stdout: "require_osd_release": "octopus", 2020-08-24T00:56:36.923 INFO:tasks.workunit.client.0.smithi060.stdout: "pools": [ 2020-08-24T00:56:36.923 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.923 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:56:36.924 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_name": "test", 2020-08-24T00:56:36.924 INFO:tasks.workunit.client.0.smithi060.stdout: "create_time": "2020-08-24T00:55:10.013555+0000", 2020-08-24T00:56:36.924 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": 1, 2020-08-24T00:56:36.924 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_names": "hashpspool", 2020-08-24T00:56:36.924 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:56:36.924 INFO:tasks.workunit.client.0.smithi060.stdout: "size": 3, 2020-08-24T00:56:36.925 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 2, 2020-08-24T00:56:36.925 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_rule": 0, 2020-08-24T00:56:36.925 INFO:tasks.workunit.client.0.smithi060.stdout: "object_hash": 2, 2020-08-24T00:56:36.925 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_autoscale_mode": "off", 2020-08-24T00:56:36.925 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num": 1, 2020-08-24T00:56:36.926 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num": 1, 2020-08-24T00:56:36.926 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num_target": 1, 2020-08-24T00:56:36.927 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_target": 1, 2020-08-24T00:56:36.928 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_pending": 1, 2020-08-24T00:56:36.929 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pg_merge_meta": { 2020-08-24T00:56:36.929 INFO:tasks.workunit.client.0.smithi060.stdout: "source_pgid": "0.0", 2020-08-24T00:56:36.929 INFO:tasks.workunit.client.0.smithi060.stdout: "ready_epoch": 0, 2020-08-24T00:56:36.929 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_started": 0, 2020-08-24T00:56:36.932 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": 0, 2020-08-24T00:56:36.933 INFO:tasks.workunit.client.0.smithi060.stdout: "source_version": "0'0", 2020-08-24T00:56:36.938 INFO:tasks.workunit.client.0.smithi060.stdout: "target_version": "0'0" 2020-08-24T00:56:36.938 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.938 INFO:tasks.workunit.client.0.smithi060.stdout: "last_change": "27", 2020-08-24T00:56:36.938 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend": "0", 2020-08-24T00:56:36.939 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2020-08-24T00:56:36.939 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2020-08-24T00:56:36.939 INFO:tasks.workunit.client.0.smithi060.stdout: "auid": 0, 2020-08-24T00:56:36.939 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_mode": "selfmanaged", 2020-08-24T00:56:36.940 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_seq": 0, 2020-08-24T00:56:36.940 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_epoch": 0, 2020-08-24T00:56:36.940 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_snaps": [], 2020-08-24T00:56:36.940 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps": "[]", 2020-08-24T00:56:36.940 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_bytes": 0, 2020-08-24T00:56:36.941 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_objects": 0, 2020-08-24T00:56:36.941 INFO:tasks.workunit.client.0.smithi060.stdout: "tiers": [], 2020-08-24T00:56:36.941 INFO:tasks.workunit.client.0.smithi060.stdout: "tier_of": -1, 2020-08-24T00:56:36.941 INFO:tasks.workunit.client.0.smithi060.stdout: "read_tier": -1, 2020-08-24T00:56:36.941 INFO:tasks.workunit.client.0.smithi060.stdout: "write_tier": -1, 2020-08-24T00:56:36.942 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_mode": "none", 2020-08-24T00:56:36.942 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_bytes": 0, 2020-08-24T00:56:36.942 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_objects": 0, 2020-08-24T00:56:36.943 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-08-24T00:56:36.943 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-08-24T00:56:36.943 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2020-08-24T00:56:36.943 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_flush_age": 0, 2020-08-24T00:56:36.943 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_evict_age": 0, 2020-08-24T00:56:36.944 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profile": "", 2020-08-24T00:56:36.944 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_params": { 2020-08-24T00:56:36.944 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "none" 2020-08-24T00:56:36.944 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.944 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_period": 0, 2020-08-24T00:56:36.945 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_count": 0, 2020-08-24T00:56:36.945 INFO:tasks.workunit.client.0.smithi060.stdout: "use_gmt_hitset": true, 2020-08-24T00:56:36.945 INFO:tasks.workunit.client.0.smithi060.stdout: "min_read_recency_for_promote": 0, 2020-08-24T00:56:36.945 INFO:tasks.workunit.client.0.smithi060.stdout: "min_write_recency_for_promote": 0, 2020-08-24T00:56:36.946 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2020-08-24T00:56:36.946 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_search_last_n": 0, 2020-08-24T00:56:36.946 INFO:tasks.workunit.client.0.smithi060.stdout: "grade_table": [], 2020-08-24T00:56:36.946 INFO:tasks.workunit.client.0.smithi060.stdout: "stripe_width": 0, 2020-08-24T00:56:36.947 INFO:tasks.workunit.client.0.smithi060.stdout: "expected_num_objects": 0, 2020-08-24T00:56:36.947 INFO:tasks.workunit.client.0.smithi060.stdout: "fast_read": false, 2020-08-24T00:56:36.947 INFO:tasks.workunit.client.0.smithi060.stdout: "options": {}, 2020-08-24T00:56:36.947 INFO:tasks.workunit.client.0.smithi060.stdout: "application_metadata": {} 2020-08-24T00:56:36.947 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.948 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.948 INFO:tasks.workunit.client.0.smithi060.stdout: "osds": [ 2020-08-24T00:56:36.948 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.948 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:56:36.948 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "0df00c36-03db-4e00-963a-1bcde275259c", 2020-08-24T00:56:36.949 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:56:36.949 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:56:36.949 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:56:36.949 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:56:36.950 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 5, 2020-08-24T00:56:36.950 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 27, 2020-08-24T00:56:36.950 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 33, 2020-08-24T00:56:36.950 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 36, 2020-08-24T00:56:36.950 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 28, 2020-08-24T00:56:36.951 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:56:36.951 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:56:36.951 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.951 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.951 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.952 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6802", 2020-08-24T00:56:36.952 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.952 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.952 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.952 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.952 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6803", 2020-08-24T00:56:36.953 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.953 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.953 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.953 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.954 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:56:36.954 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.954 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.955 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.955 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6804", 2020-08-24T00:56:36.955 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.955 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.955 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.956 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.956 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6805", 2020-08-24T00:56:36.956 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.956 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.956 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.957 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.957 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:56:36.957 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.957 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.957 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.958 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6808", 2020-08-24T00:56:36.958 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.958 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.958 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.958 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.959 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6809", 2020-08-24T00:56:36.959 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.959 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.959 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.960 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.960 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:56:36.960 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.960 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.960 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.961 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6806", 2020-08-24T00:56:36.961 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.961 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.961 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.961 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.962 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6807", 2020-08-24T00:56:36.962 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:56:36.962 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.962 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.962 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.963 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6803/24614", 2020-08-24T00:56:36.963 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6805/24614", 2020-08-24T00:56:36.963 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6809/24614", 2020-08-24T00:56:36.963 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6807/24614", 2020-08-24T00:56:36.963 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:56:36.964 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:56:36.964 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:56:36.964 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.964 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.964 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.965 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:56:36.965 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "838fa102-f839-43c9-9f7e-b2b276ac878e", 2020-08-24T00:56:36.965 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 0, 2020-08-24T00:56:36.965 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:56:36.965 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:56:36.966 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:56:36.966 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 0, 2020-08-24T00:56:36.966 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 0, 2020-08-24T00:56:36.966 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 10, 2020-08-24T00:56:36.966 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 21, 2020-08-24T00:56:36.967 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 29, 2020-08-24T00:56:36.967 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:56:36.967 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:56:36.967 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.967 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.968 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.968 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:56:36.968 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.968 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.968 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.969 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.969 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:56:36.969 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.969 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.969 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.970 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.970 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:56:36.970 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.970 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.970 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.971 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:56:36.971 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.971 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.971 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.971 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.972 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:56:36.972 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.972 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.972 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.972 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.973 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:56:36.973 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.973 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.973 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.973 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:56:36.974 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.974 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.974 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.974 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.974 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:56:36.975 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.975 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.975 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.975 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.975 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:56:36.976 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.976 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.976 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.976 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:56:36.976 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.977 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.977 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.977 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.977 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:56:36.977 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:56:36.978 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.978 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.978 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.978 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/20229", 2020-08-24T00:56:36.978 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/20229", 2020-08-24T00:56:36.979 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/20229", 2020-08-24T00:56:36.979 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/20229", 2020-08-24T00:56:36.979 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:56:36.979 INFO:tasks.workunit.client.0.smithi060.stdout: "exists" 2020-08-24T00:56:36.979 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.980 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.980 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.980 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:56:36.980 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "4423a225-da95-4821-9d96-64b9877e85a1", 2020-08-24T00:56:36.980 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:56:36.981 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:56:36.981 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:56:36.981 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:56:36.981 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 14, 2020-08-24T00:56:36.982 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 27, 2020-08-24T00:56:36.982 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 34, 2020-08-24T00:56:36.982 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 0, 2020-08-24T00:56:36.982 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 30, 2020-08-24T00:56:36.982 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:56:36.983 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:56:36.983 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.983 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.983 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.983 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:56:36.983 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.984 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.984 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.984 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.984 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:56:36.984 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.985 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.985 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.985 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.985 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:56:36.985 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.986 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.986 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.986 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:56:36.986 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.986 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.987 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.987 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.987 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:56:36.987 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.987 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.988 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.988 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.988 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:56:36.988 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.988 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.989 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.989 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:56:36.989 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.989 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.989 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.990 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.990 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:56:36.990 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.990 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.990 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.991 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.991 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:56:36.991 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:56:36.991 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.991 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:56:36.992 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:56:36.992 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.992 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.992 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.992 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:56:36.993 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:56:36.993 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:56:36.993 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.993 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.993 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.994 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/24808", 2020-08-24T00:56:36.994 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/24808", 2020-08-24T00:56:36.994 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/24808", 2020-08-24T00:56:36.994 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/24808", 2020-08-24T00:56:36.994 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:56:36.995 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:56:36.995 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:56:36.995 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:36.995 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:36.996 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:36.996 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_xinfo": [ 2020-08-24T00:56:36.996 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.996 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:56:36.996 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:55:53.738762+0000", 2020-08-24T00:56:36.997 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:56:36.997 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:56:36.997 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:56:36.997 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:56:36.998 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:54:54.857709+0000", 2020-08-24T00:56:36.998 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:56:36.998 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:36.998 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:36.998 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:56:36.999 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:55:54.278139+0000", 2020-08-24T00:56:36.999 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:56:36.999 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:56:36.999 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:56:36.999 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:56:37.000 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:55:01.513854+0000", 2020-08-24T00:56:37.000 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:56:37.000 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.000 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.000 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:56:37.001 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:55:54.805903+0000", 2020-08-24T00:56:37.001 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:56:37.001 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:56:37.001 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:56:37.001 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:56:37.002 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:55:07.838617+0000", 2020-08-24T00:56:37.002 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:56:37.002 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.002 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.002 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap": [], 2020-08-24T00:56:37.003 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap_items": [], 2020-08-24T00:56:37.003 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_temp": [], 2020-08-24T00:56:37.003 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_temp": [], 2020-08-24T00:56:37.003 INFO:tasks.workunit.client.0.smithi060.stdout: "blacklist": {}, 2020-08-24T00:56:37.003 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profiles": { 2020-08-24T00:56:37.004 INFO:tasks.workunit.client.0.smithi060.stdout: "default": { 2020-08-24T00:56:37.004 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T00:56:37.004 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "2", 2020-08-24T00:56:37.004 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T00:56:37.004 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van" 2020-08-24T00:56:37.005 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.005 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.005 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps_queue": [], 2020-08-24T00:56:37.005 INFO:tasks.workunit.client.0.smithi060.stdout: "new_removed_snaps": [], 2020-08-24T00:56:37.005 INFO:tasks.workunit.client.0.smithi060.stdout: "new_purged_snaps": [], 2020-08-24T00:56:37.006 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_node_flags": {}, 2020-08-24T00:56:37.006 INFO:tasks.workunit.client.0.smithi060.stdout: "device_class_flags": {} 2020-08-24T00:56:37.006 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.006 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_metadata": [ 2020-08-24T00:56:37.006 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:56:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:56:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6804/24614,v1:127.0.0.1:6805/24614]", 2020-08-24T00:56:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:56:37.007 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:56:37.008 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:56:37.008 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:56:37.008 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:56:37.008 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:56:37.008 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:56:37.009 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:56:37.009 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:56:37.009 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-08-24T00:56:37.009 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:56:37.009 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:56:37.010 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:56:37.010 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:56:37.010 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:56:37.010 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:56:37.010 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:56:37.011 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:56:37.011 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:56:37.011 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:56:37.011 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:56:37.011 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:56:37.012 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:56:37.012 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:56:37.012 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:56:37.012 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6802/24614,v1:127.0.0.1:6803/24614]", 2020-08-24T00:56:37.012 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:56:37.013 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/24614,v1:127.0.0.1:6809/24614]", 2020-08-24T00:56:37.013 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/24614,v1:127.0.0.1:6807/24614]", 2020-08-24T00:56:37.013 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:56:37.013 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:56:37.013 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:56:37.014 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:56:37.014 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:56:37.014 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:56:37.014 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:56:37.014 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:56:37.015 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:56:37.015 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/0", 2020-08-24T00:56:37.015 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:56:37.015 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:56:37.015 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:56:37.016 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.016 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.016 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:56:37.016 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:56:37.016 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/20229,v1:127.0.0.1:6813/20229]", 2020-08-24T00:56:37.017 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:56:37.017 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:56:37.017 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:56:37.017 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:56:37.017 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:56:37.018 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:56:37.018 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:56:37.018 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:56:37.018 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:56:37.018 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-08-24T00:56:37.019 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:56:37.019 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:56:37.019 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:56:37.019 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:56:37.019 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:56:37.020 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:56:37.020 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:56:37.020 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:56:37.020 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:56:37.020 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:56:37.021 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:56:37.021 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:56:37.021 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:56:37.021 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:56:37.021 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:56:37.022 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/20229,v1:127.0.0.1:6811/20229]", 2020-08-24T00:56:37.022 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:56:37.022 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/20229,v1:127.0.0.1:6817/20229]", 2020-08-24T00:56:37.022 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/20229,v1:127.0.0.1:6815/20229]", 2020-08-24T00:56:37.022 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:56:37.023 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:56:37.023 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:56:37.023 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:56:37.023 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:56:37.023 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:56:37.024 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:56:37.024 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:56:37.024 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:56:37.024 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/1", 2020-08-24T00:56:37.024 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:56:37.025 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:56:37.025 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:56:37.025 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.025 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.025 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:56:37.026 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:56:37.026 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/24808,v1:127.0.0.1:6813/24808]", 2020-08-24T00:56:37.026 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:56:37.026 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:56:37.026 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:56:37.027 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:56:37.027 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:56:37.027 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:56:37.027 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:56:37.027 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:56:37.028 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:56:37.028 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-08-24T00:56:37.028 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:56:37.028 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:56:37.029 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:56:37.029 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:56:37.029 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:56:37.029 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:56:37.029 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:56:37.030 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:56:37.030 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:56:37.030 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:56:37.030 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:56:37.030 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:56:37.031 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:56:37.031 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:56:37.031 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:56:37.031 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/24808,v1:127.0.0.1:6811/24808]", 2020-08-24T00:56:37.031 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:56:37.032 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/24808,v1:127.0.0.1:6817/24808]", 2020-08-24T00:56:37.032 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/24808,v1:127.0.0.1:6815/24808]", 2020-08-24T00:56:37.032 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:56:37.032 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:56:37.032 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:56:37.033 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:56:37.033 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:56:37.033 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:56:37.033 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:56:37.033 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:56:37.034 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:56:37.034 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/2", 2020-08-24T00:56:37.034 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:56:37.034 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:56:37.034 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:56:37.035 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.035 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.035 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_clean_epochs": { 2020-08-24T00:56:37.035 INFO:tasks.workunit.client.0.smithi060.stdout: "min_last_epoch_clean": 0, 2020-08-24T00:56:37.035 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": { 2020-08-24T00:56:37.036 INFO:tasks.workunit.client.0.smithi060.stdout: "per_pool": [] 2020-08-24T00:56:37.036 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.036 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_epochs": [ 2020-08-24T00:56:37.036 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.036 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:56:37.037 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 33 2020-08-24T00:56:37.037 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.037 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.037 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:56:37.037 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 10 2020-08-24T00:56:37.038 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.038 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.038 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:56:37.038 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 34 2020-08-24T00:56:37.038 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.039 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:37.039 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.039 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_first_committed": 1, 2020-08-24T00:56:37.039 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_last_committed": 37, 2020-08-24T00:56:37.039 INFO:tasks.workunit.client.0.smithi060.stdout: "crushmap": { 2020-08-24T00:56:37.040 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": [ 2020-08-24T00:56:37.040 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.040 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:56:37.040 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.0", 2020-08-24T00:56:37.040 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:56:37.040 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.041 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.041 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:56:37.041 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.1", 2020-08-24T00:56:37.041 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:56:37.042 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.042 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.042 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:56:37.042 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.2", 2020-08-24T00:56:37.042 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:56:37.043 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.043 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.043 INFO:tasks.workunit.client.0.smithi060.stdout: "types": [ 2020-08-24T00:56:37.043 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.043 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 0, 2020-08-24T00:56:37.044 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd" 2020-08-24T00:56:37.044 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.044 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.044 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:56:37.044 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "host" 2020-08-24T00:56:37.045 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.045 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.045 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 2, 2020-08-24T00:56:37.045 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "chassis" 2020-08-24T00:56:37.045 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.046 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.046 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 3, 2020-08-24T00:56:37.046 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "rack" 2020-08-24T00:56:37.046 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.046 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.047 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 4, 2020-08-24T00:56:37.047 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "row" 2020-08-24T00:56:37.047 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.047 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.047 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 5, 2020-08-24T00:56:37.048 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pdu" 2020-08-24T00:56:37.048 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.048 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.048 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 6, 2020-08-24T00:56:37.048 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pod" 2020-08-24T00:56:37.049 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.049 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.049 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 7, 2020-08-24T00:56:37.049 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "room" 2020-08-24T00:56:37.049 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.050 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.050 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 8, 2020-08-24T00:56:37.050 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "datacenter" 2020-08-24T00:56:37.050 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.050 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.051 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 9, 2020-08-24T00:56:37.051 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "zone" 2020-08-24T00:56:37.051 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.051 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.051 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 10, 2020-08-24T00:56:37.052 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "region" 2020-08-24T00:56:37.052 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.052 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.052 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:56:37.052 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "root" 2020-08-24T00:56:37.052 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.053 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.053 INFO:tasks.workunit.client.0.smithi060.stdout: "buckets": [ 2020-08-24T00:56:37.053 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.053 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -1, 2020-08-24T00:56:37.054 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default", 2020-08-24T00:56:37.054 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:56:37.054 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:56:37.054 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:56:37.054 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:56:37.055 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:56:37.055 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:56:37.055 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.055 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:56:37.056 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:56:37.056 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:56:37.056 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.056 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:37.056 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.057 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.057 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -2, 2020-08-24T00:56:37.057 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default~hdd", 2020-08-24T00:56:37.057 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:56:37.057 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:56:37.058 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:56:37.058 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:56:37.058 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:56:37.058 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:56:37.058 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.059 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:56:37.059 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:56:37.059 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:56:37.059 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.059 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:37.060 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.060 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.060 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:56:37.060 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060", 2020-08-24T00:56:37.060 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:56:37.061 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:56:37.061 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:56:37.061 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:56:37.061 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:56:37.061 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:56:37.062 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.062 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:56:37.062 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:56:37.062 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:56:37.062 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.063 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.063 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:56:37.063 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:56:37.063 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:56:37.063 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.064 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.064 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:56:37.064 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:56:37.064 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:56:37.064 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.065 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:37.065 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.065 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.065 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:56:37.065 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060~hdd", 2020-08-24T00:56:37.066 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:56:37.066 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:56:37.066 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:56:37.066 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:56:37.066 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:56:37.067 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:56:37.067 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.067 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:56:37.067 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:56:37.067 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:56:37.068 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.068 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.068 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:56:37.068 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:56:37.068 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:56:37.069 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.069 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.069 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:56:37.069 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:56:37.069 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:56:37.070 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.077 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:37.077 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.077 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.077 INFO:tasks.workunit.client.0.smithi060.stdout: "rules": [ 2020-08-24T00:56:37.077 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.078 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 0, 2020-08-24T00:56:37.078 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "replicated_rule", 2020-08-24T00:56:37.078 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 0, 2020-08-24T00:56:37.078 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:56:37.078 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 1, 2020-08-24T00:56:37.079 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 10, 2020-08-24T00:56:37.079 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T00:56:37.079 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.079 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T00:56:37.079 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T00:56:37.080 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T00:56:37.080 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.080 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.080 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_firstn", 2020-08-24T00:56:37.080 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T00:56:37.081 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T00:56:37.081 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.081 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.081 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T00:56:37.081 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.082 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:56:37.082 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.082 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.082 INFO:tasks.workunit.client.0.smithi060.stdout: "tunables": { 2020-08-24T00:56:37.082 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_tries": 0, 2020-08-24T00:56:37.083 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_fallback_tries": 0, 2020-08-24T00:56:37.083 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_total_tries": 50, 2020-08-24T00:56:37.083 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_descend_once": 1, 2020-08-24T00:56:37.083 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_vary_r": 1, 2020-08-24T00:56:37.083 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_stable": 1, 2020-08-24T00:56:37.084 INFO:tasks.workunit.client.0.smithi060.stdout: "straw_calc_version": 1, 2020-08-24T00:56:37.084 INFO:tasks.workunit.client.0.smithi060.stdout: "allowed_bucket_algs": 54, 2020-08-24T00:56:37.084 INFO:tasks.workunit.client.0.smithi060.stdout: "profile": "jewel", 2020-08-24T00:56:37.084 INFO:tasks.workunit.client.0.smithi060.stdout: "optimal_tunables": 1, 2020-08-24T00:56:37.084 INFO:tasks.workunit.client.0.smithi060.stdout: "legacy_tunables": 0, 2020-08-24T00:56:37.085 INFO:tasks.workunit.client.0.smithi060.stdout: "minimum_required_version": "jewel", 2020-08-24T00:56:37.085 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables": 1, 2020-08-24T00:56:37.085 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables2": 1, 2020-08-24T00:56:37.085 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v2_rules": 0, 2020-08-24T00:56:37.085 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables3": 1, 2020-08-24T00:56:37.086 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v3_rules": 0, 2020-08-24T00:56:37.086 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v4_buckets": 1, 2020-08-24T00:56:37.086 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables5": 1, 2020-08-24T00:56:37.086 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v5_rules": 0 2020-08-24T00:56:37.086 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.087 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_args": {} 2020-08-24T00:56:37.087 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.087 INFO:tasks.workunit.client.0.smithi060.stdout: "fsmap": { 2020-08-24T00:56:37.087 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:56:37.087 INFO:tasks.workunit.client.0.smithi060.stdout: "default_fscid": -1, 2020-08-24T00:56:37.088 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": { 2020-08-24T00:56:37.088 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": {}, 2020-08-24T00:56:37.088 INFO:tasks.workunit.client.0.smithi060.stdout: "ro_compat": {}, 2020-08-24T00:56:37.088 INFO:tasks.workunit.client.0.smithi060.stdout: "incompat": { 2020-08-24T00:56:37.088 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_1": "base v0.20", 2020-08-24T00:56:37.089 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_2": "client writeable ranges", 2020-08-24T00:56:37.089 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_3": "default file layouts on dirs", 2020-08-24T00:56:37.089 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_4": "dir inode in separate object", 2020-08-24T00:56:37.089 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_5": "mds uses versioned encoding", 2020-08-24T00:56:37.089 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_6": "dirfrag is stored in omap", 2020-08-24T00:56:37.090 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_8": "no anchor table", 2020-08-24T00:56:37.090 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_9": "file layout v2", 2020-08-24T00:56:37.090 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_10": "snaprealm v2" 2020-08-24T00:56:37.090 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.090 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.091 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_flags": { 2020-08-24T00:56:37.091 INFO:tasks.workunit.client.0.smithi060.stdout: "enable_multiple": false, 2020-08-24T00:56:37.091 INFO:tasks.workunit.client.0.smithi060.stdout: "ever_enabled_multiple": false 2020-08-24T00:56:37.091 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.091 INFO:tasks.workunit.client.0.smithi060.stdout: "standbys": [], 2020-08-24T00:56:37.092 INFO:tasks.workunit.client.0.smithi060.stdout: "filesystems": [] 2020-08-24T00:56:37.092 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.092 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_first_committed": 1, 2020-08-24T00:56:37.092 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_last_committed": 1, 2020-08-24T00:56:37.092 INFO:tasks.workunit.client.0.smithi060.stdout: "auth": { 2020-08-24T00:56:37.093 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:56:37.093 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 5, 2020-08-24T00:56:37.093 INFO:tasks.workunit.client.0.smithi060.stdout: "num_secrets": 3 2020-08-24T00:56:37.093 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.093 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T00:56:37.094 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_active": 1, 2020-08-24T00:56:37.094 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_unknown": 0, 2020-08-24T00:56:37.094 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osd": 3, 2020-08-24T00:56:37.094 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_sum": { 2020-08-24T00:56:37.094 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:56:37.095 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61700, 2020-08-24T00:56:37.095 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:56:37.095 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:56:37.095 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:56:37.095 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:56:37.096 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:56:37.096 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:56:37.096 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:56:37.096 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:56:37.097 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:56:37.097 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:56:37.097 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:56:37.097 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:56:37.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 100, 2020-08-24T00:56:37.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 100, 2020-08-24T00:56:37.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:56:37.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:56:37.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:56:37.099 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:56:37.099 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:56:37.099 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:56:37.099 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:56:37.099 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:56:37.100 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:56:37.100 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:56:37.100 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:56:37.100 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:56:37.100 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:56:37.101 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:56:37.101 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:56:37.101 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:56:37.101 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:56:37.101 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:56:37.102 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:56:37.102 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:56:37.102 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:56:37.102 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:56:37.102 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:56:37.103 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:56:37.103 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:56:37.103 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.103 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:56:37.103 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:56:37.104 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:56:37.104 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:56:37.104 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 0, 2020-08-24T00:56:37.104 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 0, 2020-08-24T00:56:37.104 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:56:37.105 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:56:37.105 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:56:37.105 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:56:37.105 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:56:37.105 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.106 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 16, 2020-08-24T00:56:37.106 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 16, 2020-08-24T00:56:37.106 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 2, 2020-08-24T00:56:37.106 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 2, 2020-08-24T00:56:37.106 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 0 2020-08-24T00:56:37.107 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.107 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum": { 2020-08-24T00:56:37.107 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:56:37.107 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:56:37.107 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T00:56:37.108 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:56:37.108 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:56:37.108 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:56:37.108 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:56:37.108 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3178368, 2020-08-24T00:56:37.109 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 32384, 2020-08-24T00:56:37.109 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 12, 2020-08-24T00:56:37.109 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145715, 2020-08-24T00:56:37.109 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311394432, 2020-08-24T00:56:37.109 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:56:37.110 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:56:37.110 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318867898368, 2020-08-24T00:56:37.110 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:56:37.110 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 33161216, 2020-08-24T00:56:37.110 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 431322, 2020-08-24T00:56:37.111 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:56:37.111 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:56:37.111 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:56:37.111 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 12734, 2020-08-24T00:56:37.111 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221212738 2020-08-24T00:56:37.112 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.112 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:56:37.112 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:56:37.112 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:56:37.112 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T00:56:37.113 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:56:37.113 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:56:37.113 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:56:37.113 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.113 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:56:37.114 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:56:37.114 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:56:37.114 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:56:37.114 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:56:37.114 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.115 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:56:37.115 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:56:37.115 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.115 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum_by_class": { 2020-08-24T00:56:37.115 INFO:tasks.workunit.client.0.smithi060.stdout: "hdd": { 2020-08-24T00:56:37.116 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:56:37.116 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:56:37.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T00:56:37.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:56:37.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:56:37.117 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:56:37.117 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:56:37.117 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3178368, 2020-08-24T00:56:37.117 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 32384, 2020-08-24T00:56:37.117 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 12, 2020-08-24T00:56:37.118 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145715, 2020-08-24T00:56:37.118 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311394432, 2020-08-24T00:56:37.118 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:56:37.118 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:56:37.118 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318867898368, 2020-08-24T00:56:37.119 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:56:37.119 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 33161216, 2020-08-24T00:56:37.119 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 431322, 2020-08-24T00:56:37.119 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:56:37.119 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:56:37.120 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:56:37.120 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 12734, 2020-08-24T00:56:37.120 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221212738 2020-08-24T00:56:37.120 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.120 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:56:37.121 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:56:37.121 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:56:37.121 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T00:56:37.121 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:56:37.121 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:56:37.122 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:56:37.122 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.122 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:56:37.122 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:56:37.122 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:56:37.123 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:56:37.123 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:56:37.123 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.123 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:56:37.123 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:56:37.124 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.124 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.124 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_stats": [ 2020-08-24T00:56:37.124 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.124 INFO:tasks.workunit.client.0.smithi060.stdout: "poolid": 1, 2020-08-24T00:56:37.125 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T00:56:37.125 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:56:37.125 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61700, 2020-08-24T00:56:37.125 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:56:37.125 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:56:37.126 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:56:37.126 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:56:37.126 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:56:37.126 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:56:37.126 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:56:37.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:56:37.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:56:37.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:56:37.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:56:37.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:56:37.128 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 100, 2020-08-24T00:56:37.128 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 100, 2020-08-24T00:56:37.128 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:56:37.128 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:56:37.129 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:56:37.129 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:56:37.129 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:56:37.129 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:56:37.129 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:56:37.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:56:37.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:56:37.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:56:37.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:56:37.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:56:37.131 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:56:37.131 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:56:37.131 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:56:37.131 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:56:37.131 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:56:37.132 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:56:37.132 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:56:37.132 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:56:37.132 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:56:37.132 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:56:37.133 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:56:37.133 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:56:37.133 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:56:37.133 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.133 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:56:37.134 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:56:37.134 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:56:37.134 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:56:37.134 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 19529728, 2020-08-24T00:56:37.134 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 184174, 2020-08-24T00:56:37.135 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:56:37.135 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:56:37.135 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:56:37.135 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:56:37.135 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:56:37.136 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.136 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 16, 2020-08-24T00:56:37.136 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 16, 2020-08-24T00:56:37.136 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 2, 2020-08-24T00:56:37.136 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 2, 2020-08-24T00:56:37.136 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 3 2020-08-24T00:56:37.137 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.137 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.137 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_stats": [ 2020-08-24T00:56:37.137 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.138 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:56:37.138 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 141733920778 2020-08-24T00:56:37.138 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.138 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.138 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:56:37.139 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 42949672978 2020-08-24T00:56:37.139 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.139 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.139 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:56:37.139 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 146028888074 2020-08-24T00:56:37.140 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.140 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.140 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_state": [ 2020-08-24T00:56:37.140 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.140 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "active+undersized+degraded", 2020-08-24T00:56:37.140 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 1 2020-08-24T00:56:37.141 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.141 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.141 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_osd": [ 2020-08-24T00:56:37.141 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.141 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:56:37.142 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:56:37.142 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T00:56:37.142 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:56:37.143 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.143 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.143 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:56:37.143 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:56:37.143 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 0, 2020-08-24T00:56:37.144 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:56:37.144 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.144 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.144 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:56:37.144 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 1, 2020-08-24T00:56:37.145 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T00:56:37.145 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:56:37.145 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.145 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.145 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": [ 2020-08-24T00:56:37.146 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:56:37.146 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:56:37.146 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": {} 2020-08-24T00:56:37.146 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.146 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:56:37.147 INFO:tasks.workunit.client.0.smithi060.stdout: "servicemap": { 2020-08-24T00:56:37.147 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:56:37.147 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:54:47.614761+0000", 2020-08-24T00:56:37.147 INFO:tasks.workunit.client.0.smithi060.stdout: "services": {} 2020-08-24T00:56:37.147 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:56:37.148 INFO:tasks.workunit.client.0.smithi060.stdout: "paxos": { 2020-08-24T00:56:37.148 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:56:37.148 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 347, 2020-08-24T00:56:37.148 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pn": 0, 2020-08-24T00:56:37.148 INFO:tasks.workunit.client.0.smithi060.stdout: "accepted_pn": 0 2020-08-24T00:56:37.148 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:56:37.149 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:56:37.150 INFO:tasks.workunit.client.0.smithi060.stderr:report 4290577564 2020-08-24T00:56:37.150 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-08-24T00:56:37.150 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-08-24T00:56:37.151 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-08-24T00:56:37.151 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-08-24T00:56:37.151 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:56:37.151 INFO:tasks.workunit.client.0.smithi060.stdout:writing non-divergent object existing_4 2020-08-24T00:56:37.250 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:56:37.250 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-08-24T00:56:02.189010+0000 27'100 37:121 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:56:37.250 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:56:37.251 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:56:37.252 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:56:37.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-08-24T00:56:37.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:56:37.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-08-24T00:56:37.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:56:37.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:56:37.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:56:37.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:56:37.424 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:56:37.431 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:56:37.431 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:56:37.432 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:56:37.432 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:56:37.432 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:56:37.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:56:37.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:56:37.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:56:37.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:56:37.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:56:37.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:56:37.946 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:56:37.946 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:56:37.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:56:37.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:56:37.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:56:38.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920781 2020-08-24T00:56:38.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 141733920781 2020-08-24T00:56:38.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-141733920781' 2020-08-24T00:56:38.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:56:38.106 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:56:38.215 INFO:tasks.workunit.client.0.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-08-24T00:56:38.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-08-24T00:56:38.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z '' 2020-08-24T00:56:38.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: continue 2020-08-24T00:56:38.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:56:38.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:56:38.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888076 2020-08-24T00:56:38.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 146028888076 2020-08-24T00:56:38.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-141733920781 2-146028888076' 2020-08-24T00:56:38.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:56:38.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-141733920781 2020-08-24T00:56:38.377 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:56:38.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:56:38.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:56:38.379 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-141733920781 2020-08-24T00:56:38.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=141733920781 2020-08-24T00:56:38.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 141733920781' 2020-08-24T00:56:38.380 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 141733920781 2020-08-24T00:56:38.384 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:56:38.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 141733920779 -lt 141733920781 2020-08-24T00:56:38.751 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:56:39.752 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:56:39.753 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:56:40.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 141733920781 -lt 141733920781 2020-08-24T00:56:40.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:56:40.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-146028888076 2020-08-24T00:56:40.166 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:56:40.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:56:40.168 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-146028888076 2020-08-24T00:56:40.169 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:56:40.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=146028888076 2020-08-24T00:56:40.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 146028888076' 2020-08-24T00:56:40.171 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 146028888076 2020-08-24T00:56:40.171 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:56:40.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 146028888076 -lt 146028888076 2020-08-24T00:56:40.538 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:56:40.538 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:40.539 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:41.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:56:41.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:41.018 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:41.019 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:41.020 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:41.020 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:41.020 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:41.021 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:41.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:41.367 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:41.367 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:41.367 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:41.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:41.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T00:56:41.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T00:56:41.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T00:56:41.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:41.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T00:56:41.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:41.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:41.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:41.927 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:41.927 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:41.927 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:41.927 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:41.928 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:42.290 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:42.290 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:42.290 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:42.291 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:42.755 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:42.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:42.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:42.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:42.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:42.757 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:42.757 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:42.757 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:42.757 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:43.198 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:43.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:43.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-08-24T00:56:43.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:43.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-08-24T00:56:43.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:43.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:43.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:43.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:43.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:43.403 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:43.404 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:43.404 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:43.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:43.755 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:43.755 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:43.755 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:44.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:44.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:44.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:44.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:44.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:44.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:44.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:44.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:44.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:44.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:44.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:44.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-08-24T00:56:44.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:44.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-08-24T00:56:45.069 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:45.070 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:45.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:45.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:45.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:45.072 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:45.072 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:45.072 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:45.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:45.426 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:45.426 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:45.426 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:45.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:45.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:45.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:45.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:45.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:45.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:45.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:45.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:45.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:46.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:46.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:46.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-08-24T00:56:46.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:46.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-08-24T00:56:47.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:47.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:47.127 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:47.128 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:47.128 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:47.128 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:47.142 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:47.142 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:47.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:47.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:47.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:47.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:47.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:47.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:47.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:47.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:47.948 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:47.948 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:47.948 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:47.949 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:47.949 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:48.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:48.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:48.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-08-24T00:56:48.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:48.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-08-24T00:56:50.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:50.002 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:50.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:50.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:50.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:50.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:50.004 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:50.004 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:50.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:50.358 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:50.358 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:50.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:50.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:50.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:50.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:50.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:50.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:50.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:50.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:50.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:50.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:51.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:51.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:51.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-08-24T00:56:51.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:51.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-08-24T00:56:54.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:56:54.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:56:54.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:56:54.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:56:54.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:56:54.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:56:54.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:56:54.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:56:54.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:56:54.862 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:56:54.862 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:56:54.862 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:56:55.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:56:55.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:56:55.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:56:55.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:56:55.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:56:55.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:56:55.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:56:55.331 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:56:55.331 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:56:55.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:56:55.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:56:55.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-08-24T00:56:55.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:56:55.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-08-24T00:57:02.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:57:02.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:57:02.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:57:02.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:57:02.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:57:02.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:57:02.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:57:02.192 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:57:02.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:57:02.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:57:02.550 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:57:02.550 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:57:03.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:57:03.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:57:03.008 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:57:03.008 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:57:03.008 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:57:03.008 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:57:03.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:57:03.009 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:57:03.010 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:57:03.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:57:03.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:57:03.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-08-24T00:57:03.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:57:03.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-08-24T00:57:10.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:57:11.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:57:11.313 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:57:11.313 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:57:11.313 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:57:11.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:57:11.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:57:11.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:57:11.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:57:11.317 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:57:11.318 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:57:11.318 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:57:11.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T00:57:11.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:57:11.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:57:11.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:57:11.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:57:11.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:57:11.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:57:11.591 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:57:11.591 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:57:12.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:57:12.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:57:12.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-08-24T00:57:12.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-08-24T00:57:12.467 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:57:12.468 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_fingerprint": "832b028f-196d-40ef-b01c-c1eaa6abfa9d", 2020-08-24T00:57:12.468 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "15.2.4-661-g0a941e5", 2020-08-24T00:57:12.468 INFO:tasks.workunit.client.0.smithi060.stdout: "commit": "0a941e55043ccee6d35a8717bec55ff7581e67d8", 2020-08-24T00:57:12.468 INFO:tasks.workunit.client.0.smithi060.stdout: "timestamp": "2020-08-24T00:57:12.464311+0000", 2020-08-24T00:57:12.468 INFO:tasks.workunit.client.0.smithi060.stdout: "tag": "", 2020-08-24T00:57:12.469 INFO:tasks.workunit.client.0.smithi060.stdout: "health": { 2020-08-24T00:57:12.469 INFO:tasks.workunit.client.0.smithi060.stdout: "status": "HEALTH_WARN", 2020-08-24T00:57:12.469 INFO:tasks.workunit.client.0.smithi060.stdout: "checks": { 2020-08-24T00:57:12.469 INFO:tasks.workunit.client.0.smithi060.stdout: "OSDMAP_FLAGS": { 2020-08-24T00:57:12.469 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:57:12.470 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:57:12.470 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "nodown,noout,noin flag(s) set", 2020-08-24T00:57:12.470 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 17 2020-08-24T00:57:12.470 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.470 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [], 2020-08-24T00:57:12.471 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:57:12.471 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.471 INFO:tasks.workunit.client.0.smithi060.stdout: "OSD_DOWN": { 2020-08-24T00:57:12.471 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:57:12.471 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:57:12.472 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 osds down", 2020-08-24T00:57:12.472 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:57:12.472 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.472 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:57:12.472 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.473 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "osd.1 (root=default,host=smithi060) is down" 2020-08-24T00:57:12.473 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.473 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.473 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:57:12.473 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.481 INFO:tasks.workunit.client.0.smithi060.stdout: "PG_DEGRADED": { 2020-08-24T00:57:12.482 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:57:12.482 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:57:12.482 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-08-24T00:57:12.482 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 2 2020-08-24T00:57:12.483 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.483 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:57:12.483 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.483 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T00:57:12.483 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.484 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.484 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:57:12.484 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.484 INFO:tasks.workunit.client.0.smithi060.stdout: "POOL_APP_NOT_ENABLED": { 2020-08-24T00:57:12.484 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:57:12.484 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:57:12.485 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 pool(s) do not have an application enabled", 2020-08-24T00:57:12.485 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:57:12.485 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.485 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:57:12.485 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.491 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "application not enabled on pool 'test'" 2020-08-24T00:57:12.492 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.492 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.492 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-08-24T00:57:12.492 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.493 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.493 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:57:12.493 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.493 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.493 INFO:tasks.workunit.client.0.smithi060.stdout: "mutes": [] 2020-08-24T00:57:12.494 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.494 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_first_committed": 1, 2020-08-24T00:57:12.495 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_last_committed": 1, 2020-08-24T00:57:12.495 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap": { 2020-08-24T00:57:12.495 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:57:12.495 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:57:12.496 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:54:46.900407Z", 2020-08-24T00:57:12.496 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:54:46.900407Z", 2020-08-24T00:57:12.496 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release": 15, 2020-08-24T00:57:12.496 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release_name": "octopus", 2020-08-24T00:57:12.496 INFO:tasks.workunit.client.0.smithi060.stdout: "features": { 2020-08-24T00:57:12.497 INFO:tasks.workunit.client.0.smithi060.stdout: "persistent": [ 2020-08-24T00:57:12.497 INFO:tasks.workunit.client.0.smithi060.stdout: "kraken", 2020-08-24T00:57:12.497 INFO:tasks.workunit.client.0.smithi060.stdout: "luminous", 2020-08-24T00:57:12.497 INFO:tasks.workunit.client.0.smithi060.stdout: "mimic", 2020-08-24T00:57:12.497 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap-prune", 2020-08-24T00:57:12.498 INFO:tasks.workunit.client.0.smithi060.stdout: "nautilus", 2020-08-24T00:57:12.498 INFO:tasks.workunit.client.0.smithi060.stdout: "octopus" 2020-08-24T00:57:12.498 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.498 INFO:tasks.workunit.client.0.smithi060.stdout: "optional": [] 2020-08-24T00:57:12.498 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.499 INFO:tasks.workunit.client.0.smithi060.stdout: "mons": [ 2020-08-24T00:57:12.499 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.499 INFO:tasks.workunit.client.0.smithi060.stdout: "rank": 0, 2020-08-24T00:57:12.499 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "a", 2020-08-24T00:57:12.499 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:57:12.500 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.500 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.500 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.500 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115", 2020-08-24T00:57:12.500 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 0 2020-08-24T00:57:12.501 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.501 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.501 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.501 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115/0", 2020-08-24T00:57:12.501 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:7115/0", 2020-08-24T00:57:12.502 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 0, 2020-08-24T00:57:12.502 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 0 2020-08-24T00:57:12.503 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.503 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.504 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.504 INFO:tasks.workunit.client.0.smithi060.stdout: "quorum": [ 2020-08-24T00:57:12.504 INFO:tasks.workunit.client.0.smithi060.stdout: 0 2020-08-24T00:57:12.505 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.505 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap": { 2020-08-24T00:57:12.505 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 37, 2020-08-24T00:57:12.506 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:57:12.506 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:54:47.614196+0000", 2020-08-24T00:57:12.506 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:56:01.914722+0000", 2020-08-24T00:57:12.506 INFO:tasks.workunit.client.0.smithi060.stdout: "last_up_change": "2020-08-24T00:56:00.174103+0000", 2020-08-24T00:57:12.507 INFO:tasks.workunit.client.0.smithi060.stdout: "last_in_change": "2020-08-24T00:56:01.605181+0000", 2020-08-24T00:57:12.507 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-08-24T00:57:12.507 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_num": 5800384, 2020-08-24T00:57:12.507 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_set": [ 2020-08-24T00:57:12.508 INFO:tasks.workunit.client.0.smithi060.stdout: "nodown", 2020-08-24T00:57:12.508 INFO:tasks.workunit.client.0.smithi060.stdout: "noin", 2020-08-24T00:57:12.508 INFO:tasks.workunit.client.0.smithi060.stdout: "noout", 2020-08-24T00:57:12.508 INFO:tasks.workunit.client.0.smithi060.stdout: "pglog_hardlimit", 2020-08-24T00:57:12.509 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snapdirs", 2020-08-24T00:57:12.509 INFO:tasks.workunit.client.0.smithi060.stdout: "recovery_deletes", 2020-08-24T00:57:12.509 INFO:tasks.workunit.client.0.smithi060.stdout: "sortbitwise" 2020-08-24T00:57:12.509 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.510 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_version": 7, 2020-08-24T00:57:12.510 INFO:tasks.workunit.client.0.smithi060.stdout: "full_ratio": 0.99000000953674316, 2020-08-24T00:57:12.510 INFO:tasks.workunit.client.0.smithi060.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-08-24T00:57:12.511 INFO:tasks.workunit.client.0.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2020-08-24T00:57:12.512 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_snapshot": "", 2020-08-24T00:57:12.512 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_max": 1, 2020-08-24T00:57:12.513 INFO:tasks.workunit.client.0.smithi060.stdout: "max_osd": 3, 2020-08-24T00:57:12.514 INFO:tasks.workunit.client.0.smithi060.stdout: "require_min_compat_client": "jewel", 2020-08-24T00:57:12.514 INFO:tasks.workunit.client.0.smithi060.stdout: "min_compat_client": "jewel", 2020-08-24T00:57:12.514 INFO:tasks.workunit.client.0.smithi060.stdout: "require_osd_release": "octopus", 2020-08-24T00:57:12.515 INFO:tasks.workunit.client.0.smithi060.stdout: "pools": [ 2020-08-24T00:57:12.515 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.515 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:57:12.516 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_name": "test", 2020-08-24T00:57:12.516 INFO:tasks.workunit.client.0.smithi060.stdout: "create_time": "2020-08-24T00:55:10.013555+0000", 2020-08-24T00:57:12.516 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": 1, 2020-08-24T00:57:12.516 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_names": "hashpspool", 2020-08-24T00:57:12.516 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:57:12.517 INFO:tasks.workunit.client.0.smithi060.stdout: "size": 3, 2020-08-24T00:57:12.517 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 2, 2020-08-24T00:57:12.517 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_rule": 0, 2020-08-24T00:57:12.517 INFO:tasks.workunit.client.0.smithi060.stdout: "object_hash": 2, 2020-08-24T00:57:12.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_autoscale_mode": "off", 2020-08-24T00:57:12.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num": 1, 2020-08-24T00:57:12.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num": 1, 2020-08-24T00:57:12.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num_target": 1, 2020-08-24T00:57:12.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_target": 1, 2020-08-24T00:57:12.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_pending": 1, 2020-08-24T00:57:12.519 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pg_merge_meta": { 2020-08-24T00:57:12.519 INFO:tasks.workunit.client.0.smithi060.stdout: "source_pgid": "0.0", 2020-08-24T00:57:12.519 INFO:tasks.workunit.client.0.smithi060.stdout: "ready_epoch": 0, 2020-08-24T00:57:12.519 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_started": 0, 2020-08-24T00:57:12.519 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": 0, 2020-08-24T00:57:12.520 INFO:tasks.workunit.client.0.smithi060.stdout: "source_version": "0'0", 2020-08-24T00:57:12.520 INFO:tasks.workunit.client.0.smithi060.stdout: "target_version": "0'0" 2020-08-24T00:57:12.520 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.520 INFO:tasks.workunit.client.0.smithi060.stdout: "last_change": "27", 2020-08-24T00:57:12.520 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend": "0", 2020-08-24T00:57:12.521 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2020-08-24T00:57:12.521 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2020-08-24T00:57:12.521 INFO:tasks.workunit.client.0.smithi060.stdout: "auid": 0, 2020-08-24T00:57:12.521 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_mode": "selfmanaged", 2020-08-24T00:57:12.521 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_seq": 0, 2020-08-24T00:57:12.522 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_epoch": 0, 2020-08-24T00:57:12.522 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_snaps": [], 2020-08-24T00:57:12.522 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps": "[]", 2020-08-24T00:57:12.522 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_bytes": 0, 2020-08-24T00:57:12.522 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_objects": 0, 2020-08-24T00:57:12.523 INFO:tasks.workunit.client.0.smithi060.stdout: "tiers": [], 2020-08-24T00:57:12.523 INFO:tasks.workunit.client.0.smithi060.stdout: "tier_of": -1, 2020-08-24T00:57:12.523 INFO:tasks.workunit.client.0.smithi060.stdout: "read_tier": -1, 2020-08-24T00:57:12.523 INFO:tasks.workunit.client.0.smithi060.stdout: "write_tier": -1, 2020-08-24T00:57:12.524 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_mode": "none", 2020-08-24T00:57:12.524 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_bytes": 0, 2020-08-24T00:57:12.524 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_objects": 0, 2020-08-24T00:57:12.524 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-08-24T00:57:12.525 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-08-24T00:57:12.525 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2020-08-24T00:57:12.525 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_flush_age": 0, 2020-08-24T00:57:12.525 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_evict_age": 0, 2020-08-24T00:57:12.525 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profile": "", 2020-08-24T00:57:12.526 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_params": { 2020-08-24T00:57:12.526 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "none" 2020-08-24T00:57:12.526 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.526 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_period": 0, 2020-08-24T00:57:12.526 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_count": 0, 2020-08-24T00:57:12.527 INFO:tasks.workunit.client.0.smithi060.stdout: "use_gmt_hitset": true, 2020-08-24T00:57:12.527 INFO:tasks.workunit.client.0.smithi060.stdout: "min_read_recency_for_promote": 0, 2020-08-24T00:57:12.527 INFO:tasks.workunit.client.0.smithi060.stdout: "min_write_recency_for_promote": 0, 2020-08-24T00:57:12.527 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2020-08-24T00:57:12.527 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_search_last_n": 0, 2020-08-24T00:57:12.528 INFO:tasks.workunit.client.0.smithi060.stdout: "grade_table": [], 2020-08-24T00:57:12.528 INFO:tasks.workunit.client.0.smithi060.stdout: "stripe_width": 0, 2020-08-24T00:57:12.528 INFO:tasks.workunit.client.0.smithi060.stdout: "expected_num_objects": 0, 2020-08-24T00:57:12.528 INFO:tasks.workunit.client.0.smithi060.stdout: "fast_read": false, 2020-08-24T00:57:12.528 INFO:tasks.workunit.client.0.smithi060.stdout: "options": {}, 2020-08-24T00:57:12.528 INFO:tasks.workunit.client.0.smithi060.stdout: "application_metadata": {} 2020-08-24T00:57:12.529 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.529 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.529 INFO:tasks.workunit.client.0.smithi060.stdout: "osds": [ 2020-08-24T00:57:12.529 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.529 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:57:12.530 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "0df00c36-03db-4e00-963a-1bcde275259c", 2020-08-24T00:57:12.530 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:57:12.530 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:57:12.530 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:57:12.530 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:57:12.531 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 5, 2020-08-24T00:57:12.531 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 27, 2020-08-24T00:57:12.531 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 33, 2020-08-24T00:57:12.531 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 36, 2020-08-24T00:57:12.531 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 28, 2020-08-24T00:57:12.531 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:57:12.532 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:57:12.532 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.532 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.532 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.532 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6802", 2020-08-24T00:57:12.533 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.533 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.533 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.533 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.533 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6803", 2020-08-24T00:57:12.534 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.534 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.534 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.535 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.535 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:57:12.535 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.535 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.535 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.536 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6804", 2020-08-24T00:57:12.536 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.536 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.536 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.536 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.537 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6805", 2020-08-24T00:57:12.537 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.537 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.537 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.537 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.538 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:57:12.538 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.538 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.539 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.539 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6808", 2020-08-24T00:57:12.539 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.539 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.539 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.549 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.549 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6809", 2020-08-24T00:57:12.549 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.549 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.550 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.550 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.550 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:57:12.550 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.551 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.551 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.551 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6806", 2020-08-24T00:57:12.552 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.552 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.552 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.553 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.553 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6807", 2020-08-24T00:57:12.553 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24614 2020-08-24T00:57:12.553 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.554 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.554 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.554 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6803/24614", 2020-08-24T00:57:12.555 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6805/24614", 2020-08-24T00:57:12.556 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6809/24614", 2020-08-24T00:57:12.556 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6807/24614", 2020-08-24T00:57:12.557 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:57:12.557 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:57:12.557 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:57:12.558 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.558 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.558 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.559 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:57:12.559 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "838fa102-f839-43c9-9f7e-b2b276ac878e", 2020-08-24T00:57:12.559 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 0, 2020-08-24T00:57:12.559 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:57:12.560 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:57:12.560 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:57:12.560 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 0, 2020-08-24T00:57:12.561 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 0, 2020-08-24T00:57:12.561 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 10, 2020-08-24T00:57:12.561 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 21, 2020-08-24T00:57:12.562 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 29, 2020-08-24T00:57:12.562 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:57:12.562 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:57:12.562 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.563 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.563 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.563 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:57:12.563 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.563 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.564 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.564 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.564 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:57:12.564 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.564 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.565 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.565 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.565 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:57:12.565 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.565 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.566 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.566 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:57:12.567 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.567 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.568 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.568 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.568 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:57:12.568 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.568 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.569 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.569 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.569 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:57:12.569 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.570 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.570 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.570 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:57:12.570 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.570 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.571 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.571 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.571 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:57:12.571 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.571 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.572 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.573 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.573 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:57:12.573 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.573 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.574 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.574 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:57:12.574 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.574 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.574 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.575 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.575 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:57:12.575 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 20229 2020-08-24T00:57:12.576 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.576 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.576 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.576 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/20229", 2020-08-24T00:57:12.576 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/20229", 2020-08-24T00:57:12.577 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/20229", 2020-08-24T00:57:12.577 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/20229", 2020-08-24T00:57:12.577 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:57:12.577 INFO:tasks.workunit.client.0.smithi060.stdout: "exists" 2020-08-24T00:57:12.577 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.578 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.578 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.578 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:57:12.578 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "4423a225-da95-4821-9d96-64b9877e85a1", 2020-08-24T00:57:12.578 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:57:12.579 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:57:12.579 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:57:12.580 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:57:12.580 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 14, 2020-08-24T00:57:12.581 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 27, 2020-08-24T00:57:12.581 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 34, 2020-08-24T00:57:12.581 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 0, 2020-08-24T00:57:12.581 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 30, 2020-08-24T00:57:12.581 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:57:12.582 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:57:12.582 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.582 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.582 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.582 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:57:12.583 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.583 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.583 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.583 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.583 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:57:12.583 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.584 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.584 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.584 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.584 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:57:12.584 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.585 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.585 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.585 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:57:12.585 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.585 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.586 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.586 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.586 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:57:12.586 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.586 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.587 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.587 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.588 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:57:12.588 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.588 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.588 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.589 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:57:12.589 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.590 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.590 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.591 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.591 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:57:12.591 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.591 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.592 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.592 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.592 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:57:12.592 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:57:12.592 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.593 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:57:12.593 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:57:12.593 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.593 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.594 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.594 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:57:12.594 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:57:12.594 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 24808 2020-08-24T00:57:12.594 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.595 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.595 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.595 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/24808", 2020-08-24T00:57:12.595 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/24808", 2020-08-24T00:57:12.595 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/24808", 2020-08-24T00:57:12.596 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/24808", 2020-08-24T00:57:12.596 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:57:12.596 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:57:12.596 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:57:12.597 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.597 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.597 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.597 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_xinfo": [ 2020-08-24T00:57:12.597 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.598 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:57:12.598 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:55:53.738762+0000", 2020-08-24T00:57:12.598 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:57:12.598 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:57:12.599 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:57:12.599 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:57:12.599 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:54:54.857709+0000", 2020-08-24T00:57:12.600 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:57:12.600 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.601 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.601 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:57:12.601 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:55:54.278139+0000", 2020-08-24T00:57:12.601 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:57:12.601 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:57:12.602 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:57:12.602 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:57:12.602 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:55:01.513854+0000", 2020-08-24T00:57:12.602 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:57:12.602 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.603 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.603 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:57:12.603 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:55:54.805903+0000", 2020-08-24T00:57:12.603 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:57:12.603 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:57:12.604 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:57:12.604 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:57:12.604 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:55:07.838617+0000", 2020-08-24T00:57:12.604 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:57:12.604 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.605 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.605 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap": [], 2020-08-24T00:57:12.605 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap_items": [], 2020-08-24T00:57:12.605 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_temp": [], 2020-08-24T00:57:12.605 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_temp": [], 2020-08-24T00:57:12.606 INFO:tasks.workunit.client.0.smithi060.stdout: "blacklist": {}, 2020-08-24T00:57:12.606 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profiles": { 2020-08-24T00:57:12.606 INFO:tasks.workunit.client.0.smithi060.stdout: "default": { 2020-08-24T00:57:12.606 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T00:57:12.606 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "2", 2020-08-24T00:57:12.607 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T00:57:12.607 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van" 2020-08-24T00:57:12.607 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.607 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.608 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps_queue": [], 2020-08-24T00:57:12.608 INFO:tasks.workunit.client.0.smithi060.stdout: "new_removed_snaps": [], 2020-08-24T00:57:12.608 INFO:tasks.workunit.client.0.smithi060.stdout: "new_purged_snaps": [], 2020-08-24T00:57:12.608 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_node_flags": {}, 2020-08-24T00:57:12.608 INFO:tasks.workunit.client.0.smithi060.stdout: "device_class_flags": {} 2020-08-24T00:57:12.609 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.609 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_metadata": [ 2020-08-24T00:57:12.609 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.609 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:57:12.610 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:57:12.611 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6804/24614,v1:127.0.0.1:6805/24614]", 2020-08-24T00:57:12.611 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:57:12.611 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:57:12.612 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:57:12.612 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:57:12.612 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:57:12.613 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:57:12.613 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:57:12.613 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:57:12.614 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:57:12.614 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-08-24T00:57:12.615 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:57:12.615 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:57:12.615 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:57:12.616 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:57:12.616 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:57:12.616 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:57:12.617 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:57:12.617 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:57:12.617 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:57:12.618 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:57:12.618 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:57:12.618 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:57:12.619 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:57:12.619 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:57:12.619 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:57:12.622 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6802/24614,v1:127.0.0.1:6803/24614]", 2020-08-24T00:57:12.622 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:57:12.623 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/24614,v1:127.0.0.1:6809/24614]", 2020-08-24T00:57:12.623 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/24614,v1:127.0.0.1:6807/24614]", 2020-08-24T00:57:12.623 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:57:12.623 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:57:12.624 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:57:12.624 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:57:12.624 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:57:12.625 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:57:12.625 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:57:12.625 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:57:12.627 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:57:12.627 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/0", 2020-08-24T00:57:12.627 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:57:12.628 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:57:12.628 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:57:12.628 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.629 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.629 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:57:12.629 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:57:12.629 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/20229,v1:127.0.0.1:6813/20229]", 2020-08-24T00:57:12.629 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:57:12.636 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:57:12.637 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:57:12.637 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:57:12.637 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:57:12.637 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:57:12.638 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:57:12.638 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:57:12.638 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:57:12.638 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-08-24T00:57:12.638 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:57:12.639 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:57:12.639 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:57:12.639 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:57:12.639 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:57:12.639 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:57:12.640 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:57:12.640 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:57:12.640 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:57:12.640 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:57:12.640 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:57:12.641 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:57:12.641 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:57:12.641 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:57:12.641 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:57:12.642 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/20229,v1:127.0.0.1:6811/20229]", 2020-08-24T00:57:12.642 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:57:12.642 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/20229,v1:127.0.0.1:6817/20229]", 2020-08-24T00:57:12.642 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/20229,v1:127.0.0.1:6815/20229]", 2020-08-24T00:57:12.642 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:57:12.643 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:57:12.643 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:57:12.643 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:57:12.643 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:57:12.643 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:57:12.644 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:57:12.644 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:57:12.644 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:57:12.644 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/1", 2020-08-24T00:57:12.644 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:57:12.645 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:57:12.645 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:57:12.645 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.645 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.645 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:57:12.645 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:57:12.646 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/24808,v1:127.0.0.1:6813/24808]", 2020-08-24T00:57:12.646 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:57:12.646 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:57:12.646 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:57:12.646 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:57:12.647 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:57:12.647 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:57:12.647 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:57:12.647 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:57:12.647 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:57:12.648 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-08-24T00:57:12.648 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:57:12.648 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:57:12.648 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:57:12.648 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:57:12.649 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:57:12.649 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:57:12.649 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:57:12.649 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:57:12.649 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:57:12.650 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:57:12.650 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:57:12.650 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:57:12.650 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:57:12.650 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:57:12.651 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:57:12.651 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/24808,v1:127.0.0.1:6811/24808]", 2020-08-24T00:57:12.651 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:57:12.651 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/24808,v1:127.0.0.1:6817/24808]", 2020-08-24T00:57:12.652 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/24808,v1:127.0.0.1:6815/24808]", 2020-08-24T00:57:12.652 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:57:12.652 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:57:12.652 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:57:12.652 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:57:12.653 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:57:12.653 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:57:12.653 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:57:12.653 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:57:12.653 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:57:12.654 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/2", 2020-08-24T00:57:12.654 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:57:12.654 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:57:12.654 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:57:12.654 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.655 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.655 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_clean_epochs": { 2020-08-24T00:57:12.655 INFO:tasks.workunit.client.0.smithi060.stdout: "min_last_epoch_clean": 0, 2020-08-24T00:57:12.656 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": { 2020-08-24T00:57:12.656 INFO:tasks.workunit.client.0.smithi060.stdout: "per_pool": [] 2020-08-24T00:57:12.656 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.656 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_epochs": [ 2020-08-24T00:57:12.656 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.657 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:57:12.657 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 33 2020-08-24T00:57:12.657 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.657 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.657 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:57:12.658 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 10 2020-08-24T00:57:12.658 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.658 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.658 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:57:12.658 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 34 2020-08-24T00:57:12.659 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.659 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.659 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.659 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_first_committed": 1, 2020-08-24T00:57:12.659 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_last_committed": 37, 2020-08-24T00:57:12.659 INFO:tasks.workunit.client.0.smithi060.stdout: "crushmap": { 2020-08-24T00:57:12.660 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": [ 2020-08-24T00:57:12.660 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.660 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:57:12.660 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.0", 2020-08-24T00:57:12.661 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:57:12.661 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.661 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.661 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:57:12.661 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.1", 2020-08-24T00:57:12.661 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:57:12.662 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.662 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.662 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:57:12.662 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.2", 2020-08-24T00:57:12.662 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:57:12.663 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.663 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.663 INFO:tasks.workunit.client.0.smithi060.stdout: "types": [ 2020-08-24T00:57:12.663 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.663 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 0, 2020-08-24T00:57:12.664 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd" 2020-08-24T00:57:12.664 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.664 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.664 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:57:12.664 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "host" 2020-08-24T00:57:12.664 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.665 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.665 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 2, 2020-08-24T00:57:12.665 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "chassis" 2020-08-24T00:57:12.665 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.665 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.666 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 3, 2020-08-24T00:57:12.666 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "rack" 2020-08-24T00:57:12.666 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.666 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.667 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 4, 2020-08-24T00:57:12.667 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "row" 2020-08-24T00:57:12.667 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.667 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.667 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 5, 2020-08-24T00:57:12.668 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pdu" 2020-08-24T00:57:12.668 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.668 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.668 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 6, 2020-08-24T00:57:12.668 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pod" 2020-08-24T00:57:12.669 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.669 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.669 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 7, 2020-08-24T00:57:12.669 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "room" 2020-08-24T00:57:12.669 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.669 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.670 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 8, 2020-08-24T00:57:12.670 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "datacenter" 2020-08-24T00:57:12.670 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.670 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.670 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 9, 2020-08-24T00:57:12.671 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "zone" 2020-08-24T00:57:12.671 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.671 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.671 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 10, 2020-08-24T00:57:12.671 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "region" 2020-08-24T00:57:12.672 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.672 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.672 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:57:12.672 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "root" 2020-08-24T00:57:12.672 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.673 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.673 INFO:tasks.workunit.client.0.smithi060.stdout: "buckets": [ 2020-08-24T00:57:12.673 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.673 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -1, 2020-08-24T00:57:12.673 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default", 2020-08-24T00:57:12.674 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:57:12.674 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:57:12.674 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:57:12.674 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:57:12.674 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:57:12.675 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:57:12.675 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.675 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:57:12.675 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:57:12.675 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:57:12.675 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.676 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.676 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.676 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.676 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -2, 2020-08-24T00:57:12.676 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default~hdd", 2020-08-24T00:57:12.677 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:57:12.677 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:57:12.677 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:57:12.677 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:57:12.677 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:57:12.678 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:57:12.678 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.678 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:57:12.678 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:57:12.678 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:57:12.678 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.679 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.679 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.679 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.679 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:57:12.679 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060", 2020-08-24T00:57:12.680 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:57:12.680 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:57:12.680 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:57:12.680 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:57:12.681 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:57:12.681 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:57:12.681 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.681 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:57:12.681 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:57:12.682 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:57:12.682 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.682 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.682 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:57:12.682 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:57:12.682 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:57:12.683 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.683 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.683 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:57:12.683 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:57:12.683 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:57:12.684 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.684 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.684 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.684 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.684 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:57:12.685 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060~hdd", 2020-08-24T00:57:12.685 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:57:12.685 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:57:12.685 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:57:12.685 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:57:12.686 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:57:12.686 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:57:12.686 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.686 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:57:12.686 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:57:12.686 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:57:12.687 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.687 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.687 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:57:12.687 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:57:12.687 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:57:12.688 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.688 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.688 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:57:12.688 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:57:12.688 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:57:12.688 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.689 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.689 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.689 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.689 INFO:tasks.workunit.client.0.smithi060.stdout: "rules": [ 2020-08-24T00:57:12.690 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.690 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 0, 2020-08-24T00:57:12.691 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "replicated_rule", 2020-08-24T00:57:12.691 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 0, 2020-08-24T00:57:12.691 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:57:12.691 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 1, 2020-08-24T00:57:12.691 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 10, 2020-08-24T00:57:12.694 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T00:57:12.694 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.694 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T00:57:12.694 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T00:57:12.695 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T00:57:12.695 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.695 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.695 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_firstn", 2020-08-24T00:57:12.695 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T00:57:12.696 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T00:57:12.696 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.696 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.696 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T00:57:12.696 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.697 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:57:12.697 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.697 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.697 INFO:tasks.workunit.client.0.smithi060.stdout: "tunables": { 2020-08-24T00:57:12.698 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_tries": 0, 2020-08-24T00:57:12.698 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_fallback_tries": 0, 2020-08-24T00:57:12.698 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_total_tries": 50, 2020-08-24T00:57:12.698 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_descend_once": 1, 2020-08-24T00:57:12.698 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_vary_r": 1, 2020-08-24T00:57:12.699 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_stable": 1, 2020-08-24T00:57:12.699 INFO:tasks.workunit.client.0.smithi060.stdout: "straw_calc_version": 1, 2020-08-24T00:57:12.699 INFO:tasks.workunit.client.0.smithi060.stdout: "allowed_bucket_algs": 54, 2020-08-24T00:57:12.699 INFO:tasks.workunit.client.0.smithi060.stdout: "profile": "jewel", 2020-08-24T00:57:12.699 INFO:tasks.workunit.client.0.smithi060.stdout: "optimal_tunables": 1, 2020-08-24T00:57:12.700 INFO:tasks.workunit.client.0.smithi060.stdout: "legacy_tunables": 0, 2020-08-24T00:57:12.700 INFO:tasks.workunit.client.0.smithi060.stdout: "minimum_required_version": "jewel", 2020-08-24T00:57:12.700 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables": 1, 2020-08-24T00:57:12.700 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables2": 1, 2020-08-24T00:57:12.701 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v2_rules": 0, 2020-08-24T00:57:12.701 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables3": 1, 2020-08-24T00:57:12.701 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v3_rules": 0, 2020-08-24T00:57:12.701 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v4_buckets": 1, 2020-08-24T00:57:12.701 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables5": 1, 2020-08-24T00:57:12.702 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v5_rules": 0 2020-08-24T00:57:12.702 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.702 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_args": {} 2020-08-24T00:57:12.702 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.702 INFO:tasks.workunit.client.0.smithi060.stdout: "fsmap": { 2020-08-24T00:57:12.703 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:57:12.703 INFO:tasks.workunit.client.0.smithi060.stdout: "default_fscid": -1, 2020-08-24T00:57:12.703 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": { 2020-08-24T00:57:12.704 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": {}, 2020-08-24T00:57:12.705 INFO:tasks.workunit.client.0.smithi060.stdout: "ro_compat": {}, 2020-08-24T00:57:12.705 INFO:tasks.workunit.client.0.smithi060.stdout: "incompat": { 2020-08-24T00:57:12.705 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_1": "base v0.20", 2020-08-24T00:57:12.706 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_2": "client writeable ranges", 2020-08-24T00:57:12.706 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_3": "default file layouts on dirs", 2020-08-24T00:57:12.706 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_4": "dir inode in separate object", 2020-08-24T00:57:12.708 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_5": "mds uses versioned encoding", 2020-08-24T00:57:12.708 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_6": "dirfrag is stored in omap", 2020-08-24T00:57:12.708 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_8": "no anchor table", 2020-08-24T00:57:12.708 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_9": "file layout v2", 2020-08-24T00:57:12.709 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_10": "snaprealm v2" 2020-08-24T00:57:12.709 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.709 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.709 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_flags": { 2020-08-24T00:57:12.710 INFO:tasks.workunit.client.0.smithi060.stdout: "enable_multiple": false, 2020-08-24T00:57:12.710 INFO:tasks.workunit.client.0.smithi060.stdout: "ever_enabled_multiple": false 2020-08-24T00:57:12.710 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.710 INFO:tasks.workunit.client.0.smithi060.stdout: "standbys": [], 2020-08-24T00:57:12.710 INFO:tasks.workunit.client.0.smithi060.stdout: "filesystems": [] 2020-08-24T00:57:12.711 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.711 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_first_committed": 1, 2020-08-24T00:57:12.711 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_last_committed": 1, 2020-08-24T00:57:12.711 INFO:tasks.workunit.client.0.smithi060.stdout: "auth": { 2020-08-24T00:57:12.711 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:57:12.712 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 5, 2020-08-24T00:57:12.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_secrets": 3 2020-08-24T00:57:12.712 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T00:57:12.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_active": 1, 2020-08-24T00:57:12.713 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_unknown": 0, 2020-08-24T00:57:12.713 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osd": 3, 2020-08-24T00:57:12.713 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_sum": { 2020-08-24T00:57:12.713 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:57:12.714 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61854, 2020-08-24T00:57:12.714 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:57:12.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:57:12.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:57:12.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:57:12.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:57:12.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:57:12.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:57:12.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:57:12.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:57:12.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:57:12.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:57:12.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:57:12.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T00:57:12.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T00:57:12.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:57:12.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:57:12.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:57:12.718 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:57:12.718 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:57:12.718 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:57:12.718 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:57:12.718 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:57:12.719 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:57:12.719 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:57:12.719 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:57:12.719 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:57:12.719 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:57:12.720 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:57:12.720 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:57:12.720 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:57:12.720 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:57:12.720 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:57:12.720 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:57:12.721 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:57:12.721 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:57:12.721 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:57:12.721 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:57:12.721 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:57:12.722 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:57:12.722 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.722 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:57:12.722 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:57:12.722 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:57:12.723 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:57:12.723 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 0, 2020-08-24T00:57:12.723 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 0, 2020-08-24T00:57:12.723 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:57:12.723 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:57:12.724 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:57:12.724 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:57:12.724 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:57:12.724 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.724 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 17, 2020-08-24T00:57:12.725 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 17, 2020-08-24T00:57:12.725 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 2, 2020-08-24T00:57:12.725 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 2, 2020-08-24T00:57:12.725 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 0 2020-08-24T00:57:12.725 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.726 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum": { 2020-08-24T00:57:12.726 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:57:12.726 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:57:12.726 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T00:57:12.726 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:57:12.726 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:57:12.727 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:57:12.727 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:57:12.727 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3178368, 2020-08-24T00:57:12.727 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 32384, 2020-08-24T00:57:12.727 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 12, 2020-08-24T00:57:12.728 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145715, 2020-08-24T00:57:12.728 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311394432, 2020-08-24T00:57:12.728 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:57:12.728 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:57:12.728 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318867898368, 2020-08-24T00:57:12.729 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:57:12.729 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 33161216, 2020-08-24T00:57:12.729 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 431630, 2020-08-24T00:57:12.729 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:57:12.729 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:57:12.729 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:57:12.730 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 12734, 2020-08-24T00:57:12.730 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221212738 2020-08-24T00:57:12.730 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.730 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:57:12.730 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:57:12.731 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:57:12.731 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T00:57:12.731 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:57:12.731 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:57:12.731 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:57:12.732 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.732 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:57:12.732 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:57:12.732 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:57:12.732 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:57:12.732 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:57:12.733 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.733 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:57:12.733 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:57:12.733 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.733 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum_by_class": { 2020-08-24T00:57:12.734 INFO:tasks.workunit.client.0.smithi060.stdout: "hdd": { 2020-08-24T00:57:12.734 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:57:12.734 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:57:12.734 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T00:57:12.734 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:57:12.735 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:57:12.735 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:57:12.735 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:57:12.735 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3178368, 2020-08-24T00:57:12.735 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 32384, 2020-08-24T00:57:12.735 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 12, 2020-08-24T00:57:12.736 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145715, 2020-08-24T00:57:12.736 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311394432, 2020-08-24T00:57:12.736 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:57:12.736 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:57:12.736 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318867898368, 2020-08-24T00:57:12.737 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:57:12.737 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 33161216, 2020-08-24T00:57:12.737 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 431630, 2020-08-24T00:57:12.738 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:57:12.738 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:57:12.738 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:57:12.738 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 12734, 2020-08-24T00:57:12.738 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221212738 2020-08-24T00:57:12.739 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.739 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:57:12.739 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:57:12.739 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:57:12.739 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T00:57:12.740 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:57:12.740 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:57:12.740 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:57:12.740 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.740 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:57:12.740 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:57:12.741 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:57:12.741 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:57:12.741 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:57:12.741 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.741 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:57:12.742 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:57:12.742 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.742 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.742 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_stats": [ 2020-08-24T00:57:12.742 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.743 INFO:tasks.workunit.client.0.smithi060.stdout: "poolid": 1, 2020-08-24T00:57:12.744 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T00:57:12.744 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:57:12.744 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61854, 2020-08-24T00:57:12.744 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:57:12.745 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:57:12.745 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:57:12.745 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:57:12.745 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:57:12.745 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:57:12.746 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:57:12.746 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:57:12.746 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:57:12.746 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:57:12.746 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:57:12.746 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:57:12.747 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T00:57:12.747 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T00:57:12.747 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:57:12.747 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:57:12.747 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:57:12.748 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:57:12.748 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:57:12.748 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:57:12.748 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:57:12.748 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:57:12.749 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:57:12.749 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:57:12.749 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:57:12.749 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:57:12.749 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:57:12.750 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:57:12.750 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:57:12.750 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:57:12.750 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:57:12.750 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:57:12.750 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:57:12.751 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:57:12.751 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:57:12.751 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:57:12.751 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:57:12.751 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:57:12.752 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:57:12.752 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.752 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:57:12.752 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:57:12.753 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:57:12.753 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:57:12.753 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 19529728, 2020-08-24T00:57:12.753 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 184482, 2020-08-24T00:57:12.753 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:57:12.754 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:57:12.754 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:57:12.754 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:57:12.754 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:57:12.754 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.755 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 17, 2020-08-24T00:57:12.755 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 17, 2020-08-24T00:57:12.755 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 2, 2020-08-24T00:57:12.755 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 2, 2020-08-24T00:57:12.755 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 3 2020-08-24T00:57:12.756 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.756 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.756 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_stats": [ 2020-08-24T00:57:12.757 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.757 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:57:12.757 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 141733920787 2020-08-24T00:57:12.757 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.757 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.758 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:57:12.758 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 42949672978 2020-08-24T00:57:12.758 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.758 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.758 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:57:12.758 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 146028888083 2020-08-24T00:57:12.759 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.759 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.759 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_state": [ 2020-08-24T00:57:12.759 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.759 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "active+undersized+degraded", 2020-08-24T00:57:12.760 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 1 2020-08-24T00:57:12.760 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.760 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.760 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_osd": [ 2020-08-24T00:57:12.760 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.761 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:57:12.761 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:57:12.761 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T00:57:12.761 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:57:12.761 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.762 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.762 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:57:12.762 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:57:12.762 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 0, 2020-08-24T00:57:12.762 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:57:12.762 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.763 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.763 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:57:12.763 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 1, 2020-08-24T00:57:12.763 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T00:57:12.763 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:57:12.764 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.764 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.764 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": [ 2020-08-24T00:57:12.764 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:57:12.764 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:57:12.764 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": {} 2020-08-24T00:57:12.765 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.765 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:57:12.765 INFO:tasks.workunit.client.0.smithi060.stdout: "servicemap": { 2020-08-24T00:57:12.765 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:57:12.765 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:54:47.614761+0000", 2020-08-24T00:57:12.766 INFO:tasks.workunit.client.0.smithi060.stdout: "services": {} 2020-08-24T00:57:12.766 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:57:12.766 INFO:tasks.workunit.client.0.smithi060.stdout: "paxos": { 2020-08-24T00:57:12.766 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:57:12.766 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 439, 2020-08-24T00:57:12.767 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pn": 0, 2020-08-24T00:57:12.767 INFO:tasks.workunit.client.0.smithi060.stdout: "accepted_pn": 0 2020-08-24T00:57:12.767 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:57:12.767 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:57:12.768 INFO:tasks.workunit.client.0.smithi060.stderr:report 751703800 2020-08-24T00:57:12.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-08-24T00:57:12.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-08-24T00:57:12.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:57:12.769 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery 2020-08-24T00:57:12.819 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:57:12.819 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-08-24T00:56:02.189010+0000 37'101 37:122 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:57:12.820 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:57:12.820 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:57:12.821 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:57:12.830 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:57:12.831 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-08-24T00:57:12.831 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T00:57:12.831 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T00:57:12.832 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:57:12.833 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:57:12.833 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:57:12.841 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T00:57:12.841 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-08-24T00:57:12.841 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok set_recovery_delay 100000 2020-08-24T00:57:12.926 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T00:57:12.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-08-24T00:57:12.934 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-08-24T00:57:12.935 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T00:57:12.935 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T00:57:12.936 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:57:12.936 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:57:12.936 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:57:12.936 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T00:57:12.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-08-24T00:57:12.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok set_recovery_delay 100000 2020-08-24T00:57:13.024 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T00:57:13.032 INFO:tasks.workunit.client.0.smithi060.stdout:revive divergent 1 2020-08-24T00:57:13.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-08-24T00:57:13.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:57:13.369 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:57:13.370 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-08-24T00:56:02.189010+0000 37'101 37:122 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:57:13.370 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:57:13.370 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:57:13.371 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:57:13.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-08-24T00:57:14.172 INFO:tasks.workunit.client.0.smithi060.stderr:noup is set 2020-08-24T00:57:14.181 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-08-24T00:57:14.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:57:14.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:57:14.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T00:57:14.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:57:14.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T00:57:14.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:57:14.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:57:14.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:57:14.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:57:14.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T00:57:14.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T00:57:14.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:57:14.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:57:14.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:57:14.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:57:14.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:57:14.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:57:14.187 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:57:14.188 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:57:14.188 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:57:14.188 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:57:14.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:57:14.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:57:14.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:57:14.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:57:14.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:57:14.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:57:14.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:57:14.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:57:14.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:57:14.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T00:57:14.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T00:57:14.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:57:14.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T00:57:14.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T00:57:14.192 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:57:14.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:57:14.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:57:14.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:57:14.226 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:14.216+0000 7f2d203f2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:57:14.226 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:14.216+0000 7f2d203f2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:57:14.227 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:14.220+0000 7f2d203f2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:57:14.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-08-24T00:57:14.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-08-24T00:57:14.758 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:14.752+0000 7f2d203f2d80 -1 Falling back to public interface 2020-08-24T00:57:15.649 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:15.644+0000 7f2d203f2d80 -1 osd.1 27 log_to_monitors {default=true} 2020-08-24T00:57:19.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-08-24T00:57:19.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:57:19.554 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery divergent 2020-08-24T00:57:19.924 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:57:19.924 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-08-24T00:56:02.189010+0000 37'101 37:123 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:57:19.925 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:57:19.925 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:57:19.925 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:57:19.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-08-24T00:57:19.933 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T00:57:19.933 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T00:57:19.933 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:57:19.934 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:57:19.934 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:57:19.934 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T00:57:19.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-08-24T00:57:19.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.1.asok set_recovery_delay 100000 2020-08-24T00:57:20.042 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T00:57:20.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-08-24T00:57:20.735 INFO:tasks.workunit.client.0.smithi060.stderr:noup is unset 2020-08-24T00:57:20.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-08-24T00:57:20.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:57:20.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:57:20.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:57:20.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:57:20.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:57:20.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:57:20.745 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:57:20.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:57:20.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:57:21.155 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 33 up_thru 36 down_at 28 last_clean_interval [5,27) [v2:127.0.0.1:6802/24614,v1:127.0.0.1:6803/24614] [v2:127.0.0.1:6804/24614,v1:127.0.0.1:6805/24614] exists,up 0df00c36-03db-4e00-963a-1bcde275259c 2020-08-24T00:57:21.156 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:57:21.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:57:21.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:57:21.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:57:21.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-08-24T00:57:21.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:57:21.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:57:21.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:57:21.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:57:21.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:57:21.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:57:21.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:57:21.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:57:21.518 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/27281,v1:127.0.0.1:6819/27281] [v2:127.0.0.1:6820/27281,v1:127.0.0.1:6821/27281] exists,up 838fa102-f839-43c9-9f7e-b2b276ac878e 2020-08-24T00:57:21.519 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:57:21.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:57:21.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:57:21.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:57:21.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-08-24T00:57:21.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:57:21.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:57:21.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:57:21.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:57:21.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:57:21.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:57:21.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:57:21.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:57:21.865 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 34 up_thru 0 down_at 30 last_clean_interval [14,27) [v2:127.0.0.1:6810/24808,v1:127.0.0.1:6811/24808] [v2:127.0.0.1:6812/24808,v1:127.0.0.1:6813/24808] exists,up 4423a225-da95-4821-9d96-64b9877e85a1 2020-08-24T00:57:21.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:57:21.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:57:21.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:57:21.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:57:22.224 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:57:22.224 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-08-24T00:56:02.189010+0000 37'101 37:123 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:57:22.225 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:57:22.225 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:57:22.226 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:57:22.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-08-24T00:57:22.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:57:22.235 INFO:tasks.workunit.client.0.smithi060.stdout:wait for peering 2020-08-24T00:57:22.578 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:57:22.578 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61854 0 0 17 17 active+undersized+degraded 2020-08-24T00:56:02.189010+0000 37'101 37:123 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:57:22.578 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:57:22.578 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:57:22.579 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:57:22.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-08-24T00:57:30.291 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-08-24T00:57:30.292 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:57:30.292 INFO:tasks.workunit.client.0.smithi060.stdout:killing divergent 1 2020-08-24T00:57:30.652 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:57:30.653 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 4 4 0 0 61854 0 0 11 11 active+recovering+degraded+wait 2020-08-24T00:57:21.291436+0000 37'101 42:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:57:30.653 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:57:30.653 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:57:30.655 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:57:30.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-08-24T00:57:30.664 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:57:30.664 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:57:30.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:57:30.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:57:30.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:57:30.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:57:30.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.11819.out 2020-08-24T00:57:30.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.11819.out 2020-08-24T00:57:30.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-08-24T00:57:30.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T00:57:30.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-08-24T00:57:30.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T00:57:30.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-08-24T00:57:30.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.11819.out 2020-08-24T00:57:31.393 INFO:tasks.workunit.client.0.smithi060.stderr:Exporting 1.0 info 1.0( v 42'102 lc 0'0 (27'90,42'102] local-lis/les=41/42 n=101 ec=21/21 lis/c=41/21 les/c/f=42/22/0 sis=41 pruub=8.902248569s) 2020-08-24T00:57:31.405 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:023dd3d2:::existing_12:head# 2020-08-24T00:57:31.411 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:04c40a0c:::existing_93:head# 2020-08-24T00:57:31.429 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:064ff5fb:::existing_36:head# 2020-08-24T00:57:31.450 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:091f9f1d:::existing_94:head# 2020-08-24T00:57:31.451 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:0a865a93:::existing_54:head# 2020-08-24T00:57:31.458 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:0fed513c:::existing_41:head# 2020-08-24T00:57:31.459 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-08-24T00:57:31.462 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:169157fa:::existing_81:head# 2020-08-24T00:57:31.466 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:17731382:::existing_58:head# 2020-08-24T00:57:31.467 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:18a45a39:::existing_43:head# 2020-08-24T00:57:31.468 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:1c160f01:::existing_66:head# 2020-08-24T00:57:31.470 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:1dd6c249:::existing_97:head# 2020-08-24T00:57:31.471 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:227e0c77:::existing_7:head# 2020-08-24T00:57:31.472 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:23e1b063:::existing_59:head# 2020-08-24T00:57:31.473 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:28566edb:::existing_78:head# 2020-08-24T00:57:31.482 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:2a101182:::existing_20:head# 2020-08-24T00:57:31.488 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:2d1519da:::existing_96:head# 2020-08-24T00:57:31.488 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:3078b87a:::existing_86:head# 2020-08-24T00:57:31.489 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:30bde01b:::existing_15:head# 2020-08-24T00:57:31.490 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:317a7e19:::existing_56:head# 2020-08-24T00:57:31.490 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-08-24T00:57:31.492 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:3e16d84b:::existing_48:head# 2020-08-24T00:57:31.493 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:45fe8203:::existing_75:head# 2020-08-24T00:57:31.497 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-08-24T00:57:31.498 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:5298dcdb:::existing_57:head# 2020-08-24T00:57:31.499 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:52ebaa44:::existing_32:head# 2020-08-24T00:57:31.499 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:53588912:::existing_9:head# 2020-08-24T00:57:31.500 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-08-24T00:57:31.509 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:5da0255c:::existing_13:head# 2020-08-24T00:57:31.510 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:5da2dab8:::existing_53:head# 2020-08-24T00:57:31.511 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:5ee3514b:::existing_30:head# 2020-08-24T00:57:31.512 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:602f83fe:::foo:head# 2020-08-24T00:57:31.512 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:60426cc7:::existing_62:head# 2020-08-24T00:57:31.519 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:628918ca:::existing_79:head# 2020-08-24T00:57:31.519 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-08-24T00:57:31.520 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:67e1780f:::existing_27:head# 2020-08-24T00:57:31.521 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:68143ca1:::existing_71:head# 2020-08-24T00:57:31.531 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:6bac9509:::existing_29:head# 2020-08-24T00:57:31.531 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:6bbec798:::existing_45:head# 2020-08-24T00:57:31.532 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:6cb23a40:::existing_65:head# 2020-08-24T00:57:31.532 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:7055ec5e:::existing_83:head# 2020-08-24T00:57:31.533 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:70a54e2b:::existing_87:head# 2020-08-24T00:57:31.534 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:750ace21:::existing_19:head# 2020-08-24T00:57:31.534 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:78623b1d:::existing_6:head# 2020-08-24T00:57:31.535 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:79881152:::existing_63:head# 2020-08-24T00:57:31.535 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-08-24T00:57:31.539 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:81490677:::existing_76:head# 2020-08-24T00:57:31.540 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-08-24T00:57:31.540 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-08-24T00:57:31.541 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-08-24T00:57:31.542 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:923c5d4d:::existing_5:head# 2020-08-24T00:57:31.542 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:92731e71:::existing_89:head# 2020-08-24T00:57:31.543 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:928205cf:::existing_82:head# 2020-08-24T00:57:31.544 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:93be2267:::existing_24:head# 2020-08-24T00:57:31.544 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:99ceda2c:::existing_44:head# 2020-08-24T00:57:31.545 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:9dd917ee:::existing_26:head# 2020-08-24T00:57:31.545 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:a28805a6:::existing_72:head# 2020-08-24T00:57:31.546 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:a294a8f6:::existing_16:head# 2020-08-24T00:57:31.546 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:a2f5473a:::existing_42:head# 2020-08-24T00:57:31.547 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:a504a533:::existing_90:head# 2020-08-24T00:57:31.548 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:a691337e:::existing_61:head# 2020-08-24T00:57:31.548 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-08-24T00:57:31.549 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:a9fde86e:::existing_18:head# 2020-08-24T00:57:31.553 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:aa309faa:::existing_68:head# 2020-08-24T00:57:31.554 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-08-24T00:57:31.554 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:b279b1fe:::existing_10:head# 2020-08-24T00:57:31.555 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:b343e277:::existing_88:head# 2020-08-24T00:57:31.556 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:b7130e90:::existing_80:head# 2020-08-24T00:57:31.557 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:bc6f7309:::existing_70:head# 2020-08-24T00:57:31.558 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:be8489b7:::existing_38:head# 2020-08-24T00:57:31.558 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-08-24T00:57:31.559 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:bebec877:::existing_74:head# 2020-08-24T00:57:31.559 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:c04347d8:::existing_60:head# 2020-08-24T00:57:31.560 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:c156033e:::existing_100:head# 2020-08-24T00:57:31.560 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-08-24T00:57:31.561 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-08-24T00:57:31.562 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-08-24T00:57:31.563 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-08-24T00:57:31.564 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:ccb3865b:::existing_73:head# 2020-08-24T00:57:31.564 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:cd84e19e:::existing_40:head# 2020-08-24T00:57:31.565 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-08-24T00:57:31.565 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:d41a25c9:::existing_22:head# 2020-08-24T00:57:31.566 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:d7f088da:::existing_91:head# 2020-08-24T00:57:31.567 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-08-24T00:57:31.567 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-08-24T00:57:31.568 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:dd677bd3:::existing_35:head# 2020-08-24T00:57:31.568 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-08-24T00:57:31.569 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:e0e22077:::existing_21:head# 2020-08-24T00:57:31.569 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:e255b710:::existing_55:head# 2020-08-24T00:57:31.570 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:eff1d5db:::existing_14:head# 2020-08-24T00:57:31.584 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:f1b156bf:::existing_23:head# 2020-08-24T00:57:31.592 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:f5ce4199:::existing_34:head# 2020-08-24T00:57:31.593 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:f6809456:::existing_11:head# 2020-08-24T00:57:31.609 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:f699a2a9:::existing_67:head# 2020-08-24T00:57:31.610 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:f9baac43:::existing_25:head# 2020-08-24T00:57:31.610 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:fb76688e:::existing_51:head# 2020-08-24T00:57:31.611 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:fbb92eec:::existing_69:head# 2020-08-24T00:57:31.612 INFO:tasks.workunit.client.0.smithi060.stderr:Export successful 2020-08-24T00:57:31.612 INFO:tasks.workunit.client.0.smithi060.stdout: marking collection for removal 2020-08-24T00:57:31.612 INFO:tasks.workunit.client.0.smithi060.stdout:setting '_remove' omap key 2020-08-24T00:57:31.613 INFO:tasks.workunit.client.0.smithi060.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-08-24T00:57:31.872 INFO:tasks.workunit.client.0.smithi060.stderr:Remove successful 2020-08-24T00:57:32.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.11819.out 2020-08-24T00:57:32.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-08-24T00:57:32.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T00:57:32.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-08-24T00:57:32.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T00:57:32.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-08-24T00:57:32.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.11819.out 2020-08-24T00:57:32.911 INFO:tasks.workunit.client.0.smithi060.stdout:get_pg_num_history pg_num_history pg_num_history(e42 pg_nums {1={21=1}} deleted_pools ) 2020-08-24T00:57:32.912 INFO:tasks.workunit.client.0.smithi060.stdout:Importing pgid 1.0 2020-08-24T00:57:32.912 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:023dd3d2:::existing_12:head# 2020-08-24T00:57:32.913 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.011 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:04c40a0c:::existing_93:head# 2020-08-24T00:57:33.012 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.063 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:064ff5fb:::existing_36:head# 2020-08-24T00:57:33.063 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.117 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:091f9f1d:::existing_94:head# 2020-08-24T00:57:33.117 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.150 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:0a865a93:::existing_54:head# 2020-08-24T00:57:33.151 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.192 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:0fed513c:::existing_41:head# 2020-08-24T00:57:33.193 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.244 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-08-24T00:57:33.244 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.294 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:169157fa:::existing_81:head# 2020-08-24T00:57:33.295 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.381 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:17731382:::existing_58:head# 2020-08-24T00:57:33.382 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.440 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:18a45a39:::existing_43:head# 2020-08-24T00:57:33.440 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.512 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:1c160f01:::existing_66:head# 2020-08-24T00:57:33.512 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.559 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:1dd6c249:::existing_97:head# 2020-08-24T00:57:33.559 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.601 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:227e0c77:::existing_7:head# 2020-08-24T00:57:33.602 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.643 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:23e1b063:::existing_59:head# 2020-08-24T00:57:33.644 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.669 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:28566edb:::existing_78:head# 2020-08-24T00:57:33.669 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.694 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:2a101182:::existing_20:head# 2020-08-24T00:57:33.694 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.720 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:2d1519da:::existing_96:head# 2020-08-24T00:57:33.720 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.753 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:3078b87a:::existing_86:head# 2020-08-24T00:57:33.753 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.787 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:30bde01b:::existing_15:head# 2020-08-24T00:57:33.787 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.823 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:317a7e19:::existing_56:head# 2020-08-24T00:57:33.824 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.858 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-08-24T00:57:33.858 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.891 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:3e16d84b:::existing_48:head# 2020-08-24T00:57:33.891 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.924 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:45fe8203:::existing_75:head# 2020-08-24T00:57:33.925 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.958 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-08-24T00:57:33.958 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:33.991 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:5298dcdb:::existing_57:head# 2020-08-24T00:57:33.992 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.026 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:52ebaa44:::existing_32:head# 2020-08-24T00:57:34.026 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.059 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:53588912:::existing_9:head# 2020-08-24T00:57:34.060 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.102 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-08-24T00:57:34.103 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.130 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:5da0255c:::existing_13:head# 2020-08-24T00:57:34.131 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.155 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:5da2dab8:::existing_53:head# 2020-08-24T00:57:34.155 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.178 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:5ee3514b:::existing_30:head# 2020-08-24T00:57:34.178 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.204 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:602f83fe:::foo:head# 2020-08-24T00:57:34.204 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.230 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:60426cc7:::existing_62:head# 2020-08-24T00:57:34.234 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.258 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:628918ca:::existing_79:head# 2020-08-24T00:57:34.258 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.292 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-08-24T00:57:34.292 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.325 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:67e1780f:::existing_27:head# 2020-08-24T00:57:34.325 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.358 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:68143ca1:::existing_71:head# 2020-08-24T00:57:34.359 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.378 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:6bac9509:::existing_29:head# 2020-08-24T00:57:34.378 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.402 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:6bbec798:::existing_45:head# 2020-08-24T00:57:34.402 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.430 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:6cb23a40:::existing_65:head# 2020-08-24T00:57:34.430 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.452 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:7055ec5e:::existing_83:head# 2020-08-24T00:57:34.452 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.477 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:70a54e2b:::existing_87:head# 2020-08-24T00:57:34.478 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.510 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:750ace21:::existing_19:head# 2020-08-24T00:57:34.510 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.528 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:78623b1d:::existing_6:head# 2020-08-24T00:57:34.529 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.562 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:79881152:::existing_63:head# 2020-08-24T00:57:34.562 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.590 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-08-24T00:57:34.590 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.647 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:81490677:::existing_76:head# 2020-08-24T00:57:34.648 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.672 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-08-24T00:57:34.673 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.699 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-08-24T00:57:34.699 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.726 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-08-24T00:57:34.726 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.757 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:923c5d4d:::existing_5:head# 2020-08-24T00:57:34.757 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.790 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:92731e71:::existing_89:head# 2020-08-24T00:57:34.791 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.829 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:928205cf:::existing_82:head# 2020-08-24T00:57:34.830 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.858 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:93be2267:::existing_24:head# 2020-08-24T00:57:34.859 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.892 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:99ceda2c:::existing_44:head# 2020-08-24T00:57:34.892 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.925 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:9dd917ee:::existing_26:head# 2020-08-24T00:57:34.925 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.960 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:a28805a6:::existing_72:head# 2020-08-24T00:57:34.961 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:34.986 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:a294a8f6:::existing_16:head# 2020-08-24T00:57:34.986 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.014 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:a2f5473a:::existing_42:head# 2020-08-24T00:57:35.014 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.036 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:a504a533:::existing_90:head# 2020-08-24T00:57:35.037 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.062 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:a691337e:::existing_61:head# 2020-08-24T00:57:35.062 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.088 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-08-24T00:57:35.089 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.112 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:a9fde86e:::existing_18:head# 2020-08-24T00:57:35.113 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.138 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:aa309faa:::existing_68:head# 2020-08-24T00:57:35.138 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.173 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-08-24T00:57:35.173 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.264 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:b279b1fe:::existing_10:head# 2020-08-24T00:57:35.264 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.300 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:b343e277:::existing_88:head# 2020-08-24T00:57:35.300 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.335 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:b7130e90:::existing_80:head# 2020-08-24T00:57:35.336 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.369 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:bc6f7309:::existing_70:head# 2020-08-24T00:57:35.370 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.401 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:be8489b7:::existing_38:head# 2020-08-24T00:57:35.402 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.435 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-08-24T00:57:35.435 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.460 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:bebec877:::existing_74:head# 2020-08-24T00:57:35.460 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.485 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:c04347d8:::existing_60:head# 2020-08-24T00:57:35.486 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.513 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:c156033e:::existing_100:head# 2020-08-24T00:57:35.513 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.538 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-08-24T00:57:35.539 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.563 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-08-24T00:57:35.563 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.589 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-08-24T00:57:35.590 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.614 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-08-24T00:57:35.614 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.640 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:ccb3865b:::existing_73:head# 2020-08-24T00:57:35.640 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.664 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:cd84e19e:::existing_40:head# 2020-08-24T00:57:35.664 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.689 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-08-24T00:57:35.690 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.715 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:d41a25c9:::existing_22:head# 2020-08-24T00:57:35.716 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.741 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:d7f088da:::existing_91:head# 2020-08-24T00:57:35.741 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.774 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-08-24T00:57:35.774 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.811 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-08-24T00:57:35.811 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.842 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:dd677bd3:::existing_35:head# 2020-08-24T00:57:35.843 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.875 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-08-24T00:57:35.875 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.909 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:e0e22077:::existing_21:head# 2020-08-24T00:57:35.909 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.944 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:e255b710:::existing_55:head# 2020-08-24T00:57:35.945 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.969 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:eff1d5db:::existing_14:head# 2020-08-24T00:57:35.970 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:35.995 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:f1b156bf:::existing_23:head# 2020-08-24T00:57:35.995 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:36.033 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:f5ce4199:::existing_34:head# 2020-08-24T00:57:36.033 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:36.045 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:f6809456:::existing_11:head# 2020-08-24T00:57:36.046 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:36.114 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:f699a2a9:::existing_67:head# 2020-08-24T00:57:36.115 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:36.138 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:f9baac43:::existing_25:head# 2020-08-24T00:57:36.138 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:36.164 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:fb76688e:::existing_51:head# 2020-08-24T00:57:36.164 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:36.190 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:fbb92eec:::existing_69:head# 2020-08-24T00:57:36.191 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T00:57:36.214 INFO:tasks.workunit.client.0.smithi060.stdout:write_pg epoch 42 info 1.0( v 42'102 lc 0'0 (27'90,42'102] local-lis/les=41/42 n=101 ec=21/21 lis/c=41/21 les/c/f=42/22/0 sis=41 pruub=8.902248569s) 2020-08-24T00:57:36.248 INFO:tasks.workunit.client.0.smithi060.stdout:Import successful 2020-08-24T00:57:36.790 INFO:tasks.workunit.client.0.smithi060.stdout:reviving divergent 1 2020-08-24T00:57:36.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-08-24T00:57:36.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:57:37.151 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:57:37.151 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 4 4 0 0 62471 0 0 12 12 active+recovering+degraded 2020-08-24T00:57:30.192440+0000 42'102 42:128 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:57:37.151 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:57:37.152 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:57:37.154 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:57:37.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-08-24T00:57:37.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:57:37.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:57:37.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T00:57:37.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:57:37.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T00:57:37.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:57:37.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:57:37.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:57:37.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:57:37.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T00:57:37.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T00:57:37.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:57:37.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:57:37.168 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:57:37.168 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:57:37.168 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:57:37.168 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:57:37.171 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:57:37.172 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:57:37.172 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:57:37.172 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:57:37.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:57:37.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:57:37.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:57:37.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:57:37.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:57:37.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:57:37.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:57:37.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:57:37.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:57:37.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T00:57:37.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T00:57:37.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:57:37.184 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T00:57:37.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T00:57:37.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:57:37.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:57:37.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:57:37.185 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:57:37.222 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:37.216+0000 7ff97abd2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:57:37.226 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:37.216+0000 7ff97abd2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:57:37.226 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:37.220+0000 7ff97abd2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:57:37.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-08-24T00:57:37.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:57:37.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:57:37.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:57:37.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:57:37.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:57:37.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:57:37.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:57:37.553 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:57:37.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:57:37.757 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:37.752+0000 7ff97abd2d80 -1 Falling back to public interface 2020-08-24T00:57:37.939 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/27281,v1:127.0.0.1:6819/27281] [v2:127.0.0.1:6820/27281,v1:127.0.0.1:6821/27281] exists,up 838fa102-f839-43c9-9f7e-b2b276ac878e 2020-08-24T00:57:37.939 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:57:37.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:57:37.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:57:37.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:57:37.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-08-24T00:57:37.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:57:37.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:57:37.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:57:37.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:57:37.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:57:37.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:57:37.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:57:37.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:57:38.326 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 41 up_thru 41 down_at 29 last_clean_interval [10,27) [v2:127.0.0.1:6818/27281,v1:127.0.0.1:6819/27281] [v2:127.0.0.1:6820/27281,v1:127.0.0.1:6821/27281] exists,up 838fa102-f839-43c9-9f7e-b2b276ac878e 2020-08-24T00:57:38.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:57:38.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:57:38.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:57:38.328 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-08-24T00:57:38.736 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:57:38.732+0000 7ff97abd2d80 -1 osd.1 42 log_to_monitors {default=true} 2020-08-24T00:57:58.332 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-08-24T00:58:22.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T00:58:22.423 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T00:58:22.423 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:58:22.423 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.423 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.424 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T00:58:22.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-08-24T00:58:22.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.1.asok dump_ops_in_flight 2020-08-24T00:58:22.464 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:58:22.465 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [], 2020-08-24T00:58:22.465 INFO:tasks.workunit.client.0.smithi060.stdout: "num_ops": 0 2020-08-24T00:58:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:58:22.465 INFO:tasks.workunit.client.0.smithi060.stdout:allowing recovery 2020-08-24T00:58:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:58:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 0 0 0 62471 0 0 12 12 active+clean 2020-08-24T00:57:40.552903+0000 42'102 45:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:58:22.466 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:58:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:58:22.466 INFO:tasks.workunit.client.0.smithi060.stdout:reading divergent objects 2020-08-24T00:58:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:58:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 0 0 0 62471 0 0 12 12 active+clean 2020-08-24T00:57:40.552903+0000 42'102 45:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:55:10.063737+0000 0'0 2020-08-24T00:55:10.063737+0000 0 2020-08-24T00:58:22.467 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:58:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:58:22.467 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 27'106 (27'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-08-24T00:55:42.942680+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T00:58:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: keeping 27'106 (27'4) delete 1:70b3bf12:::existing_4:head by client.4541.0:1 2020-08-24T00:55:42.942680+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T00:58:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T00:58:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 27'4 first_divergent_update: 27'106 last_divergent_update: 27'106 2020-08-24T00:58:22.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: more recent entry found: 37'101 (27'4) modify 1:70b3bf12:::existing_4:head by client.4650.0:1 2020-08-24T00:56:37.304425+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0, already merged 2020-08-24T00:58:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 27'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4529.0:1 2020-08-24T00:55:42.931362+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: keeping 27'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4529.0:1 2020-08-24T00:55:42.931362+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T00:58:22.469 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 27'102 last_divergent_update: 27'102 2020-08-24T00:58:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-08-24T00:58:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-08-24T00:58:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 27'103 (27'3) delete 1:8881e6d8:::existing_3:head by client.4537.0:1 2020-08-24T00:55:42.939936+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T00:58:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: keeping 27'103 (27'3) delete 1:8881e6d8:::existing_3:head by client.4537.0:1 2020-08-24T00:55:42.939936+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T00:58:22.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-08-24T00:58:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 27'3 first_divergent_update: 27'103 last_divergent_update: 27'103 2020-08-24T00:58:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-08-24T00:58:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-08-24T00:58:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 27'106 2020-08-24T00:58:22.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 27'103 (27'3) delete 1:8881e6d8:::existing_3:head by client.4537.0:1 2020-08-24T00:55:42.939936+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T00:58:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-08-24T00:58:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 27'3 <= info.log_tail 27'90 2020-08-24T00:58:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 27'105 (27'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-08-24T00:55:42.940249+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: keeping 27'105 (27'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-08-24T00:55:42.940249+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T00:58:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 27'2 first_divergent_update: 27'105 last_divergent_update: 27'105 2020-08-24T00:58:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-08-24T00:58:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-08-24T00:58:22.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 27'106 2020-08-24T00:58:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 27'105 (27'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-08-24T00:55:42.940249+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-08-24T00:58:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 27'2 <= info.log_tail 27'90 2020-08-24T00:58:22.474 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 27'104 (27'1) modify 1:d858c035:::existing_1:head by client.4538.0:1 2020-08-24T00:55:42.939968+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: keeping 27'104 (27'1) modify 1:d858c035:::existing_1:head by client.4538.0:1 2020-08-24T00:55:42.939968+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T00:58:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 27'1 first_divergent_update: 27'104 last_divergent_update: 27'104 2020-08-24T00:58:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-08-24T00:58:22.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-08-24T00:58:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 27'106 2020-08-24T00:58:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 27'104 (27'1) modify 1:d858c035:::existing_1:head by client.4538.0:1 2020-08-24T00:55:42.939968+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-08-24T00:58:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 27'1 <= info.log_tail 27'90 2020-08-24T00:58:22.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 27'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4526.0:1 2020-08-24T00:55:42.916495+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.477 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 20 _merge_object_divergent_entries: keeping 27'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4526.0:1 2020-08-24T00:55:42.916495+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T00:58:22.477 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T00:58:22.477 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 27'101 last_divergent_update: 27'101 2020-08-24T00:58:22.478 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-08-24T00:58:22.478 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T00:57:21.128+0000 7f2cff02f700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-08-24T00:58:22.478 INFO:tasks.workunit.client.0.smithi060.stdout:success 2020-08-24T00:58:22.478 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 765fa77d-4249-474d-a0a3-ac21e9faf855 2020-08-24T00:58:22.479 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:58:22.479 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T00:58:22.479 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T00:58:22.479 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:58:22.479 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:58:22.480 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/28762,v1:127.0.0.1:6803/28762] [v2:127.0.0.1:6804/28762,v1:127.0.0.1:6805/28762] exists,up 765fa77d-4249-474d-a0a3-ac21e9faf855 2020-08-24T00:58:22.480 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 b9f5bd36-904e-4fe9-8925-445d22957285 2020-08-24T00:58:22.480 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:58:22.480 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T00:58:22.481 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T00:58:22.481 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:58:22.481 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:58:22.481 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/29117,v1:127.0.0.1:6811/29117] [v2:127.0.0.1:6812/29117,v1:127.0.0.1:6813/29117] exists,up b9f5bd36-904e-4fe9-8925-445d22957285 2020-08-24T00:58:22.481 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 03903bbb-6dd8-4f53-b407-7ee8ec7022ce 2020-08-24T00:58:22.482 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T00:58:22.482 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T00:58:22.482 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T00:58:22.482 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:58:22.482 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:58:22.483 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/29472,v1:127.0.0.1:6819/29472] [v2:127.0.0.1:6820/29472,v1:127.0.0.1:6821/29472] exists,up 03903bbb-6dd8-4f53-b407-7ee8ec7022ce 2020-08-24T00:58:22.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-08-24T00:58:22.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:58:22.483 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:58:22.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-08-24T00:58:22.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-08-24T00:58:22.484 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T00:58:22.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-08-24T00:58:22.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-08-24T00:58:22.485 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T00:58:22.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-08-24T00:58:22.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-08-24T00:58:22.485 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T00:58:22.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-08-24T00:58:22.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-08-24T00:58:22.486 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:58:22.486 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-08-24T00:58:22.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-08-24T00:58:22.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:58:22.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-08-24T00:58:22.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:58:22.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-08-24T00:58:22.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:58:22.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-08-24T00:58:22.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T00:58:22.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-08-24T00:58:22.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-08-24T00:58:22.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T00:58:22.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-08-24T00:58:22.489 INFO:tasks.workunit.client.0.smithi060.stderr:error getting test/newobject_1: (2) No such file or directory 2020-08-24T00:58:22.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T00:58:22.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-08-24T00:58:22.489 INFO:tasks.workunit.client.0.smithi060.stderr:error getting test/newobject_2: (2) No such file or directory 2020-08-24T00:58:22.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-08-24T00:58:22.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-08-24T00:58:22.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T00:58:22.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-08-24T00:58:22.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T00:58:22.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-08-24T00:58:22.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T00:58:22.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-08-24T00:58:22.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.11819.out 2020-08-24T00:58:22.492 INFO:tasks.workunit.client.0.smithi060.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.11819.out': No such file or directory 2020-08-24T00:58:22.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-08-24T00:58:22.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-08-24T00:58:22.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-08-24T00:58:22.493 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' does not exist 2020-08-24T00:58:22.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-08-24T00:58:22.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:58:22.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:58:22.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:58:22.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:58:22.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:58:22.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:58:22.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-08-24T00:58:22.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T00:58:22.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T00:58:22.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T00:58:22.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:58:22.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:58:22.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:58:22.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:58:22.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:58:22.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:58:22.496 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:58:22.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:58:22.497 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:58:22.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:58:22.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:58:22.497 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:58:22.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:58:22.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:58:22.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:58:22.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:58:22.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:58:22.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T00:58:22.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T00:58:22.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:58:22.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T00:58:22.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:58:22.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:58:22.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-08-24T00:58:22.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-08-24T00:58:22.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-08-24T00:58:22.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-08-24T00:58:22.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T00:58:22.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T00:58:22.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T00:58:22.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:58:22.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:58:22.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:58:22.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:58:22.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:58:22.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:58:22.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T00:58:22.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T00:58:22.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T00:58:22.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T00:58:22.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T00:58:22.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T00:58:22.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:58:22.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T00:58:22.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T00:58:22.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T00:58:22.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T00:58:22.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T00:58:22.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T00:58:22.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T00:58:22.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T00:58:22.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T00:58:22.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T00:58:22.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-08-24T00:58:22.507 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T00:58:22.507 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.507 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.11819 2020-08-24T00:58:22.508 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T00:58:22.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-08-24T00:58:22.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T00:58:22.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-08-24T00:58:22.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-08-24T00:58:22.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-08-24T00:58:22.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-08-24T00:58:22.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-08-24T00:58:22.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-08-24T00:58:22.510 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-08-24T00:58:22.510 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-08-24T00:58:22.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-08-24T00:58:22.510 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:58:22.511 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:58:22.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-08-24T00:58:22.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-08-24T00:58:22.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T00:58:22.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T00:58:22.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T00:58:22.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-08-24T00:58:22.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-08-24T00:58:22.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T00:58:22.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:58:22.513 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:58:22.513 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:58:22.513 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.513 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.513 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T00:58:22.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T00:58:22.514 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T00:58:22.514 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:58:22.514 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:58:22.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T00:58:22.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T00:58:22.515 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:58:22.515 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:58:22.515 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:58:22.516 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:58:22.516 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.516 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.516 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T00:58:22.516 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:58:22.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get fsid 2020-08-24T00:58:22.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T00:58:22.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T00:58:22.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T00:58:22.518 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T00:58:22.518 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T00:58:22.518 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T00:58:22.518 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T00:58:22.518 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T00:58:22.519 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:58:22.519 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.519 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.519 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T00:58:22.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T00:58:22.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get mon_host 2020-08-24T00:58:22.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-08-24T00:58:22.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-08-24T00:58:22.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T00:58:22.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T00:58:22.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T00:58:22.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-08-24T00:58:22.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T00:58:22.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T00:58:22.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:58:22.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:58:22.522 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:58:22.522 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.523 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:58:22.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T00:58:22.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-08-24T00:58:22.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-08-24T00:58:22.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:58:22.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:58:22.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T00:58:22.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:58:22.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-08-24T00:58:22.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:58:22.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:58:22.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:58:22.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:58:22.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T00:58:22.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T00:58:22.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:58:22.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:58:22.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:58:22.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:58:22.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:58:22.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:58:22.528 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:58:22.528 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.528 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.529 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:58:22.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:58:22.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:58:22.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:58:22.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:58:22.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:58:22.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:58:22.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:58:22.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:58:22.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:58:22.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-08-24T00:58:22.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:58:22.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=765fa77d-4249-474d-a0a3-ac21e9faf855 2020-08-24T00:58:22.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 765fa77d-4249-474d-a0a3-ac21e9faf855' 2020-08-24T00:58:22.532 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:58:22.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCaEENfX1A6CBAAxCkUm/FIZULIMz1nhUms8A== 2020-08-24T00:58:22.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCaEENfX1A6CBAAxCkUm/FIZULIMz1nhUms8A=="}' 2020-08-24T00:58:22.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 765fa77d-4249-474d-a0a3-ac21e9faf855 -i td/divergent-priors/0/new.json 2020-08-24T00:58:22.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-08-24T00:58:22.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCaEENfX1A6CBAAxCkUm/FIZULIMz1nhUms8A== --osd-uuid 765fa77d-4249-474d-a0a3-ac21e9faf855 2020-08-24T00:58:22.533 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.512+0000 7f257341dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.533 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.512+0000 7f257341dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.534 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.516+0000 7f257341dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.534 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.516+0000 7f257341dd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:58:22.534 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.516+0000 7f257341dd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:58:22.534 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.516+0000 7f257341dd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T00:58:22.534 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.552+0000 7f257341dd80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-08-24T00:58:22.535 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:02.992+0000 7f257341dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:58:22.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-08-24T00:58:22.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:58:22.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T00:58:22.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:58:22.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T00:58:22.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:58:22.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:58:22.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:58:22.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:58:22.537 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:06.200+0000 7f10c5605d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.537 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:06.200+0000 7f10c5605d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.537 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:06.200+0000 7f10c5605d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T00:58:22.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:58:22.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:58:22.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:58:22.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:58:22.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:58:22.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:58:22.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:58:22.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:58:22.540 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:06.740+0000 7f10c5605d80 -1 Falling back to public interface 2020-08-24T00:58:22.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:58:22.540 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:07.528+0000 7f10c5605d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T00:58:22.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:58:22.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:58:22.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:58:22.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:58:22.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:58:22.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:58:22.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:58:22.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:58:22.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-08-24T00:58:22.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-08-24T00:58:22.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:58:22.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:58:22.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T00:58:22.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:58:22.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-08-24T00:58:22.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:58:22.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:58:22.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:58:22.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:58:22.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T00:58:22.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T00:58:22.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:58:22.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:58:22.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:58:22.546 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:58:22.547 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:58:22.547 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:58:22.547 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:58:22.547 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.547 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:58:22.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:58:22.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:58:22.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:58:22.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:58:22.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:58:22.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:58:22.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:58:22.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:58:22.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:58:22.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-08-24T00:58:22.550 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:58:22.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b9f5bd36-904e-4fe9-8925-445d22957285 2020-08-24T00:58:22.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 b9f5bd36-904e-4fe9-8925-445d22957285' 2020-08-24T00:58:22.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:58:22.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCgEENfZocMFRAAUeH9n9CYUhIFx88vnTKheg== 2020-08-24T00:58:22.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCgEENfZocMFRAAUeH9n9CYUhIFx88vnTKheg=="}' 2020-08-24T00:58:22.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b9f5bd36-904e-4fe9-8925-445d22957285 -i td/divergent-priors/1/new.json 2020-08-24T00:58:22.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-08-24T00:58:22.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCgEENfZocMFRAAUeH9n9CYUhIFx88vnTKheg== --osd-uuid b9f5bd36-904e-4fe9-8925-445d22957285 2020-08-24T00:58:22.552 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:08.968+0000 7f41cca34d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.552 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:08.972+0000 7f41cca34d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.553 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:08.972+0000 7f41cca34d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.553 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:08.972+0000 7f41cca34d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:58:22.553 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:08.972+0000 7f41cca34d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:58:22.553 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:08.972+0000 7f41cca34d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T00:58:22.553 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:09.056+0000 7f41cca34d80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-08-24T00:58:22.554 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:09.460+0000 7f41cca34d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:58:22.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-08-24T00:58:22.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:58:22.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T00:58:22.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:58:22.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T00:58:22.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:58:22.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:58:22.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:58:22.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:58:22.556 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:12.768+0000 7faa3af7cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.556 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:12.772+0000 7faa3af7cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.556 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:12.772+0000 7faa3af7cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T00:58:22.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:58:22.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T00:58:22.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:58:22.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:58:22.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:58:22.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:58:22.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:58:22.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:58:22.559 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:13.316+0000 7faa3af7cd80 -1 Falling back to public interface 2020-08-24T00:58:22.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:58:22.560 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:14.104+0000 7faa3af7cd80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T00:58:22.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:58:22.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:58:22.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:58:22.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:58:22.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T00:58:22.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:58:22.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:58:22.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:58:22.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-08-24T00:58:22.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-08-24T00:58:22.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T00:58:22.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T00:58:22.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T00:58:22.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T00:58:22.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-08-24T00:58:22.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:58:22.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:58:22.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:58:22.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:58:22.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T00:58:22.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T00:58:22.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T00:58:22.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T00:58:22.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:58:22.565 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T00:58:22.565 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:58:22.565 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:58:22.567 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:58:22.567 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:22.567 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:22.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:58:22.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:58:22.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T00:58:22.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T00:58:22.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:58:22.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:58:22.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:58:22.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:58:22.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:58:22.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T00:58:22.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T00:58:22.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-08-24T00:58:22.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T00:58:22.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=03903bbb-6dd8-4f53-b407-7ee8ec7022ce 2020-08-24T00:58:22.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 03903bbb-6dd8-4f53-b407-7ee8ec7022ce' 2020-08-24T00:58:22.571 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T00:58:22.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCmEENflFstNRAA/jGktchg6vKHF0kIigMFqQ== 2020-08-24T00:58:22.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCmEENflFstNRAA/jGktchg6vKHF0kIigMFqQ=="}' 2020-08-24T00:58:22.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 03903bbb-6dd8-4f53-b407-7ee8ec7022ce -i td/divergent-priors/2/new.json 2020-08-24T00:58:22.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-08-24T00:58:22.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCmEENflFstNRAA/jGktchg6vKHF0kIigMFqQ== --osd-uuid 03903bbb-6dd8-4f53-b407-7ee8ec7022ce 2020-08-24T00:58:22.572 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:15.548+0000 7f9edd5d3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:15.548+0000 7f9edd5d3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:15.552+0000 7f9edd5d3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:15.552+0000 7f9edd5d3d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:58:22.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:15.552+0000 7f9edd5d3d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:58:22.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:15.552+0000 7f9edd5d3d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T00:58:22.574 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:15.600+0000 7f9edd5d3d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-08-24T00:58:22.574 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:16.108+0000 7f9edd5d3d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T00:58:22.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-08-24T00:58:22.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T00:58:22.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T00:58:22.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T00:58:22.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T00:58:22.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T00:58:22.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T00:58:22.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:58:22.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T00:58:22.576 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:19.540+0000 7fc99334cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.576 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:19.544+0000 7fc99334cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.576 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:19.544+0000 7fc99334cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:58:22.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T00:58:22.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:58:22.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:58:22.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:58:22.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:58:22.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:58:22.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:58:22.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:58:22.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:58:22.578 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:20.084+0000 7fc99334cd80 -1 Falling back to public interface 2020-08-24T00:58:22.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:58:22.578 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:58:20.876+0000 7fc99334cd80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T00:58:22.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:58:22.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:58:22.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:58:22.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:58:22.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:58:22.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:58:22.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:58:22.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:58:22.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-08-24T00:58:22.580 INFO:tasks.workunit.client.0.smithi060.stderr:noout is set 2020-08-24T00:58:22.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-08-24T00:58:23.062 INFO:tasks.workunit.client.0.smithi060.stderr:noin is set 2020-08-24T00:58:23.075 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-08-24T00:58:23.824 INFO:tasks.workunit.client.0.smithi060.stderr:nodown is set 2020-08-24T00:58:23.832 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-08-24T00:58:23.833 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-08-24T00:58:24.354 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T00:58:24.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T00:58:25.364 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-08-24T00:58:26.000 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 size to 3 2020-08-24T00:58:26.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-08-24T00:58:26.790 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 min_size to 2 2020-08-24T00:58:26.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-08-24T00:58:27.528 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 pg_autoscale_mode to on 2020-08-24T00:58:27.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-08-24T00:58:27.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:58:27.539 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:58:27.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:58:27.929 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:58:27.929 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:58:27.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:58:27.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:27.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:58:28.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-08-24T00:58:28.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836486 2020-08-24T00:58:28.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486' 2020-08-24T00:58:28.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:28.106 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:58:28.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-08-24T00:58:28.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672965 2020-08-24T00:58:28.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-08-24T00:58:28.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:28.267 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:58:28.431 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-08-24T00:58:28.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509444 2020-08-24T00:58:28.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486 1-42949672965 2-64424509444' 2020-08-24T00:58:28.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:28.433 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836486 2020-08-24T00:58:28.433 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:28.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:58:28.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2020-08-24T00:58:28.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:28.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836486 2020-08-24T00:58:28.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-08-24T00:58:28.438 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836486 2020-08-24T00:58:28.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:58:28.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836484 -lt 21474836486 2020-08-24T00:58:28.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:58:29.820 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:58:29.821 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:58:30.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836487 -lt 21474836486 2020-08-24T00:58:30.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:30.194 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672965 2020-08-24T00:58:30.195 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:30.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:58:30.196 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:30.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672965 2020-08-24T00:58:30.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672965 2020-08-24T00:58:30.198 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-08-24T00:58:30.198 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672965 2020-08-24T00:58:30.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:58:30.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672965 -lt 42949672965 2020-08-24T00:58:30.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:30.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509444 2020-08-24T00:58:30.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:30.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:58:30.598 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509444 2020-08-24T00:58:30.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:30.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509444 2020-08-24T00:58:30.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-08-24T00:58:30.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:58:30.603 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509444 2020-08-24T00:58:30.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509444 -lt 64424509444 2020-08-24T00:58:30.950 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-08-24T00:58:30.950 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:58:30.950 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:58:30.950 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:58:30.951 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:58:30.951 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T00:58:30.952 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:58:30.952 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:58:30.952 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:58:30.952 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:58:30.952 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:58:31.148 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:58:31.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:58:31.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:58:31.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:58:31.150 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:58:31.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:58:31.527 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:58:31.528 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:58:31.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:58:31.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:31.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:58:31.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-08-24T00:58:31.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836489 2020-08-24T00:58:31.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489' 2020-08-24T00:58:31.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:31.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:58:31.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-08-24T00:58:31.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672968 2020-08-24T00:58:31.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489 1-42949672968' 2020-08-24T00:58:31.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:31.882 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:58:32.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-08-24T00:58:32.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509446 2020-08-24T00:58:32.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836489 1-42949672968 2-64424509446' 2020-08-24T00:58:32.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:32.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836489 2020-08-24T00:58:32.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:32.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:58:32.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836489 2020-08-24T00:58:32.055 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:32.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836489 2020-08-24T00:58:32.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-08-24T00:58:32.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:58:32.058 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836489 2020-08-24T00:58:32.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836487 -lt 21474836489 2020-08-24T00:58:32.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:58:33.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:58:33.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:58:33.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836489 -lt 21474836489 2020-08-24T00:58:33.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:33.811 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672968 2020-08-24T00:58:33.812 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:33.812 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:58:33.814 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:33.814 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672968 2020-08-24T00:58:33.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672968 2020-08-24T00:58:33.815 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-08-24T00:58:33.815 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:58:33.815 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672968 2020-08-24T00:58:34.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672968 -lt 42949672968 2020-08-24T00:58:34.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:34.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:34.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509446 2020-08-24T00:58:34.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:58:34.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509446 2020-08-24T00:58:34.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:34.220 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509446 2020-08-24T00:58:34.220 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-08-24T00:58:34.221 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:58:34.221 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509446 2020-08-24T00:58:34.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509447 -lt 64424509446 2020-08-24T00:58:34.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:58:34.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:58:34.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:58:35.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:58:35.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:58:35.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:58:35.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:58:35.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:58:35.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:58:35.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:58:35.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:58:35.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T00:58:35.415 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:58:35.415 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:58:35.415 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:58:35.872 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T00:58:35.872 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T00:58:35.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T00:58:35.873 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-08-24T00:58:35.873 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-08-24T00:58:36.221 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:58:36.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-08-24T00:58:36.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-08-24T00:58:36.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-08-24T00:58:36.229 INFO:tasks.workunit.client.0.smithi060.stdout:primary and soon to be divergent is 1 2020-08-24T00:58:36.574 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:58:36.575 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T00:58:25.119799+0000 0'0 24:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:58:36.575 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:58:36.575 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:58:36.575 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:58:36.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-08-24T00:58:36.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-08-24T00:58:36.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-08-24T00:58:36.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-08-24T00:58:36.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-08-24T00:58:36.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-08-24T00:58:36.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-08-24T00:58:36.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-08-24T00:58:36.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-08-24T00:58:36.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-08-24T00:58:36.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-08-24T00:58:36.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-08-24T00:58:36.580 INFO:tasks.workunit.client.0.smithi060.stdout:writing initial objects 2020-08-24T00:58:36.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:36.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-08-24T00:58:36.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:36.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-08-24T00:58:36.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:36.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-08-24T00:58:37.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:37.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-08-24T00:58:37.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:37.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-08-24T00:58:37.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:37.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-08-24T00:58:37.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:37.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-08-24T00:58:37.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:37.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-08-24T00:58:37.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:37.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-08-24T00:58:37.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:37.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-08-24T00:58:38.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:38.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-08-24T00:58:38.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:38.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-08-24T00:58:38.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:38.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-08-24T00:58:38.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:38.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-08-24T00:58:38.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:38.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-08-24T00:58:38.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:38.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-08-24T00:58:38.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:38.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-08-24T00:58:39.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:39.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-08-24T00:58:39.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:39.263 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-08-24T00:58:39.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:39.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-08-24T00:58:39.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:39.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-08-24T00:58:39.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:39.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-08-24T00:58:39.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:39.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-08-24T00:58:39.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:39.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-08-24T00:58:40.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:40.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-08-24T00:58:40.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:40.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-08-24T00:58:40.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:40.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-08-24T00:58:40.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:40.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-08-24T00:58:40.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:40.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-08-24T00:58:40.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:40.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-08-24T00:58:41.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:41.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-08-24T00:58:41.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:41.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-08-24T00:58:41.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:41.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-08-24T00:58:41.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:41.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-08-24T00:58:41.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:41.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-08-24T00:58:41.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:41.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-08-24T00:58:41.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:41.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-08-24T00:58:42.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:42.143 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-08-24T00:58:42.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:42.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-08-24T00:58:42.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:42.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-08-24T00:58:42.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:42.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-08-24T00:58:42.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:42.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-08-24T00:58:42.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:42.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-08-24T00:58:43.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:43.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-08-24T00:58:43.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:43.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-08-24T00:58:43.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:43.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-08-24T00:58:43.444 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:43.444 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-08-24T00:58:43.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:43.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-08-24T00:58:43.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:43.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-08-24T00:58:43.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:43.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-08-24T00:58:44.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:44.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-08-24T00:58:44.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:44.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-08-24T00:58:44.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:44.336 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-08-24T00:58:44.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:44.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-08-24T00:58:44.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:44.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-08-24T00:58:44.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:44.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-08-24T00:58:44.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:44.962 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-08-24T00:58:45.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:45.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-08-24T00:58:45.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:45.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-08-24T00:58:45.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:45.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-08-24T00:58:45.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:45.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-08-24T00:58:45.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:45.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-08-24T00:58:45.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:45.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-08-24T00:58:45.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:45.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-08-24T00:58:46.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:46.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-08-24T00:58:46.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:46.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-08-24T00:58:46.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:46.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-08-24T00:58:46.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:46.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-08-24T00:58:46.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:46.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-08-24T00:58:46.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:46.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-08-24T00:58:47.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:47.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-08-24T00:58:47.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:47.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-08-24T00:58:47.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:47.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-08-24T00:58:47.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:47.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-08-24T00:58:47.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:47.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-08-24T00:58:47.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:47.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-08-24T00:58:47.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:47.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-08-24T00:58:48.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:48.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-08-24T00:58:48.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:48.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-08-24T00:58:48.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:48.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-08-24T00:58:48.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:48.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-08-24T00:58:48.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:48.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-08-24T00:58:48.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:48.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-08-24T00:58:48.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:48.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-08-24T00:58:49.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:49.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-08-24T00:58:49.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:49.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-08-24T00:58:49.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:49.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-08-24T00:58:49.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:49.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-08-24T00:58:49.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:49.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-08-24T00:58:49.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:49.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-08-24T00:58:49.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:49.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-08-24T00:58:50.139 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:50.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-08-24T00:58:50.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:50.332 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-08-24T00:58:50.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:50.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-08-24T00:58:50.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:50.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-08-24T00:58:50.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:50.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-08-24T00:58:50.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:50.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-08-24T00:58:51.068 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:51.069 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-08-24T00:58:51.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:51.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-08-24T00:58:51.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-08-24T00:58:51.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-08-24T00:58:51.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:58:51.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-08-24T00:58:51.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:58:51.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:58:51.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:58:51.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:58:51.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:58:51.483 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:58:51.483 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:58:51.483 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:58:51.484 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:58:51.484 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:58:51.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:58:51.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:58:51.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:58:51.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:58:51.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:58:51.995 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:58:51.995 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:58:51.995 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:58:51.996 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:58:51.996 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:51.996 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:58:52.176 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836495 2020-08-24T00:58:52.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836495 2020-08-24T00:58:52.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495' 2020-08-24T00:58:52.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:52.178 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:58:52.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672974 2020-08-24T00:58:52.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672974 2020-08-24T00:58:52.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672974' 2020-08-24T00:58:52.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:58:52.354 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:58:52.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509453 2020-08-24T00:58:52.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509453 2020-08-24T00:58:52.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672974 2-64424509453' 2020-08-24T00:58:52.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:52.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836495 2020-08-24T00:58:52.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:52.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:58:52.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2020-08-24T00:58:52.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:52.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836495 2020-08-24T00:58:52.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-08-24T00:58:52.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:58:52.531 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836495 2020-08-24T00:58:52.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836493 -lt 21474836495 2020-08-24T00:58:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:58:53.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:58:53.898 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:58:54.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836495 -lt 21474836495 2020-08-24T00:58:54.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:54.302 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672974 2020-08-24T00:58:54.303 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:54.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T00:58:54.305 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672974 2020-08-24T00:58:54.305 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:54.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672974 2020-08-24T00:58:54.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672974' 2020-08-24T00:58:54.308 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672974 2020-08-24T00:58:54.309 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T00:58:54.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672974 -lt 42949672974 2020-08-24T00:58:54.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:58:54.671 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509453 2020-08-24T00:58:54.671 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:58:54.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:58:54.672 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509453 2020-08-24T00:58:54.672 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:58:54.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509453 2020-08-24T00:58:54.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509453' 2020-08-24T00:58:54.673 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:58:54.674 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509453 2020-08-24T00:58:55.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509453 -lt 64424509453 2020-08-24T00:58:55.041 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:58:55.041 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:58:55.041 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:58:55.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T00:58:58.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:58:58.613 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:58:58.613 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:58:58.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:58:58.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:58:58.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:58:58.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:58:58.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T00:58:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:58:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:58:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:58:58.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T00:58:58.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T00:58:58.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T00:58:58.616 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-08-24T00:58:58.616 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-08-24T00:58:58.616 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-08-24T00:58:58.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-08-24T00:58:58.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-08-24T00:58:58.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-08-24T00:58:58.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-08-24T00:58:58.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-08-24T00:58:58.630 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:58:58.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:58:58.631 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-08-24T00:58:58.631 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T00:58:58.631 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T00:58:58.631 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:58:58.631 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:58.632 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:58.632 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T00:58:58.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-08-24T00:58:58.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok config set objectstore_blackhole 1 2020-08-24T00:58:58.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:58:58.633 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-08-24T00:58:58.633 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T00:58:58.633 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T00:58:58.633 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T00:58:58.633 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:58:58.634 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:58:58.634 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T00:58:58.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-08-24T00:58:58.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok config set objectstore_blackhole 1 2020-08-24T00:58:58.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-08-24T00:58:58.635 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-08-24T00:58:58.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T00:58:58.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T00:58:58.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-08-24T00:58:58.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-08-24T00:58:58.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-08-24T00:58:58.636 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-08-24T00:58:58.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-08-24T00:58:58.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-08-24T00:58:58.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-08-24T00:58:58.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-08-24T00:58:58.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-08-24T00:58:58.637 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-08-24T00:58:58.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-08-24T00:58:58.637 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-08-24T00:58:58.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-08-24T00:58:58.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-08-24T00:58:58.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-08-24T00:58:58.640 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-08-24T00:58:58.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-08-24T00:58:58.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-08-24T00:58:58.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-08-24T00:58:58.641 INFO:tasks.workunit.client.0.smithi060.stdout:blackholing osds 0 2 2020-08-24T00:58:58.641 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:58:58.642 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-08-24T00:58:25.119799+0000 30'100 30:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:58:58.642 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:58:58.642 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:58:58.642 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:58:58.642 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T00:58:58.643 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:58:58.643 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:58:58.643 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T00:58:58.643 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:58:58.643 INFO:tasks.workunit.client.0.smithi060.stdout:create new divergent objects 2020-08-24T00:58:58.643 INFO:tasks.workunit.client.0.smithi060.stdout:writing divergent objects 2020-08-24T00:58:58.644 INFO:tasks.workunit.client.0.smithi060.stdout:remove divergent objects 2020-08-24T00:59:07.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-08-24T00:59:07.217 INFO:tasks.workunit.client.0.smithi060.stderr:rados: no process found 2020-08-24T00:59:07.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-08-24T00:59:07.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-08-24T00:59:07.218 INFO:tasks.workunit.client.0.smithi060.stdout:killing all the osds 2020-08-24T00:59:07.581 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:59:07.582 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 active+clean 2020-08-24T00:58:25.119799+0000 30'106 30:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:07.582 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:59:07.583 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:59:07.584 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:59:07.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-08-24T00:59:07.594 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T00:59:07.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T00:59:07.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T00:59:07.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T00:59:07.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T00:59:07.707 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T00:59:07.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-08-24T00:59:07.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-08-24T00:59:08.291 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already down. 2020-08-24T00:59:08.300 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-08-24T00:59:08.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-08-24T00:59:08.805 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already down. 2020-08-24T00:59:08.815 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-08-24T00:59:08.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-08-24T00:59:09.336 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already down. 2020-08-24T00:59:09.345 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:59:09.345 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-08-24T00:59:09.878 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already out. 2020-08-24T00:59:09.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:59:09.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-08-24T00:59:10.424 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already out. 2020-08-24T00:59:10.436 INFO:tasks.workunit.client.0.smithi060.stdout:bringing up non_divergent 0 2 2020-08-24T00:59:10.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-08-24T00:59:10.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-08-24T00:59:10.779 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:59:10.779 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.780 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.780 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.780 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.780 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.781 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.781 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.781 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.781 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.781 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.782 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.782 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.782 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.782 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.782 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.783 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.783 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.783 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.783 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.784 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.784 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 60774 0 0 16 16 stale+active+clean 2020-08-24T00:58:25.119799+0000 30'106 30:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:10.784 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.784 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.784 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.785 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.785 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.785 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.785 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.785 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.786 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.786 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.786 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 0 0 0 0 0 0 0 0 0 0 unknown 2020-08-24T00:59:07.655867+0000 0'0 0:0 [] -1 [] -1 0'0 2020-08-24T00:59:07.655867+0000 0'0 2020-08-24T00:59:07.655867+0000 0 2020-08-24T00:59:10.786 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:59:10.787 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:59:10.787 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:59:10.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:59:10.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-08-24T00:59:10.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:59:10.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:59:10.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-08-24T00:59:10.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:59:10.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-08-24T00:59:10.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:59:10.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:59:10.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:59:10.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:59:10.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T00:59:10.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T00:59:10.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:59:10.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:59:10.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:59:10.795 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:59:10.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:59:10.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:59:10.796 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:59:10.796 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:59:10.796 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:59:10.797 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:59:10.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:59:10.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:59:10.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:59:10.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:59:10.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:59:10.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:59:10.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:59:10.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:59:10.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:59:10.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-08-24T00:59:10.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-08-24T00:59:10.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:59:10.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-08-24T00:59:10.800 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T00:59:10.801 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-08-24T00:59:10.801 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:59:10.801 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:59:10.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:59:10.823 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:10.815+0000 7f1c7276ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:59:10.830 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:10.819+0000 7f1c7276ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:59:10.830 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:10.819+0000 7f1c7276ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:59:11.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-08-24T00:59:11.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:59:11.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T00:59:11.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:59:11.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:59:11.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:59:11.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:59:11.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:59:11.167 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:59:11.168 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:59:11.361 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:11.355+0000 7f1c7276ad80 -1 Falling back to public interface 2020-08-24T00:59:11.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:59:12.281 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:12.275+0000 7f1c7276ad80 -1 osd.0 30 log_to_monitors {default=true} 2020-08-24T00:59:12.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:59:12.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:59:12.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:59:12.543 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:59:12.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:59:12.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T00:59:12.921 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up out weight 0 up_from 38 up_thru 0 down_at 33 last_clean_interval [5,30) [v2:127.0.0.1:6802/33546,v1:127.0.0.1:6803/33546] [v2:127.0.0.1:6804/33546,v1:127.0.0.1:6805/33546] exists,up 765fa77d-4249-474d-a0a3-ac21e9faf855 2020-08-24T00:59:12.921 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T00:59:12.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:59:12.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:59:12.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:59:12.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:59:12.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-08-24T00:59:12.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T00:59:12.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T00:59:12.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-08-24T00:59:12.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T00:59:12.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-08-24T00:59:12.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T00:59:12.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T00:59:12.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T00:59:12.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T00:59:12.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T00:59:12.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T00:59:12.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T00:59:12.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T00:59:12.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T00:59:12.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T00:59:12.927 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T00:59:12.927 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T00:59:12.927 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T00:59:12.927 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T00:59:12.927 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T00:59:12.928 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:59:12.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T00:59:12.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T00:59:12.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T00:59:12.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T00:59:12.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T00:59:12.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T00:59:12.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:59:12.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T00:59:12.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T00:59:12.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-08-24T00:59:12.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-08-24T00:59:12.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T00:59:12.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-08-24T00:59:12.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-08-24T00:59:12.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T00:59:12.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T00:59:12.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T00:59:12.947 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:12.939+0000 7fd66becad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:59:12.950 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:12.943+0000 7fd66becad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:59:12.951 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:12.943+0000 7fd66becad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T00:59:13.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-08-24T00:59:13.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T00:59:13.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T00:59:13.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T00:59:13.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T00:59:13.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:59:13.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T00:59:13.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:59:13.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:59:13.310 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T00:59:13.489 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:13.483+0000 7fd66becad80 -1 Falling back to public interface 2020-08-24T00:59:13.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T00:59:14.493 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T00:59:14.487+0000 7fd66becad80 -1 osd.2 30 log_to_monitors {default=true} 2020-08-24T00:59:14.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T00:59:14.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T00:59:14.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T00:59:14.677 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T00:59:14.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T00:59:14.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T00:59:15.050 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up out weight 0 up_from 39 up_thru 0 down_at 35 last_clean_interval [15,30) [v2:127.0.0.1:6810/33740,v1:127.0.0.1:6811/33740] [v2:127.0.0.1:6812/33740,v1:127.0.0.1:6813/33740] exists,up 03903bbb-6dd8-4f53-b407-7ee8ec7022ce 2020-08-24T00:59:15.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T00:59:15.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T00:59:15.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T00:59:15.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:59:15.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-08-24T00:59:15.588 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already in. 2020-08-24T00:59:15.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-08-24T00:59:15.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-08-24T00:59:16.180 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already in. 2020-08-24T00:59:16.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:59:16.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-08-24T00:59:16.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:59:16.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:59:16.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:59:16.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:59:16.195 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:59:16.196 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:59:16.196 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:59:16.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:59:16.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:59:16.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:59:16.286 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:59:16.286 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:59:16.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:59:16.289 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:59:16.290 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:59:16.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:59:16.697 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:59:16.697 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:59:16.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:59:16.698 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:59:16.698 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:59:16.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=163208757251 2020-08-24T00:59:16.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 163208757251 2020-08-24T00:59:16.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-163208757251' 2020-08-24T00:59:16.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:59:16.864 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:59:16.983 INFO:tasks.workunit.client.0.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-08-24T00:59:16.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-08-24T00:59:16.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z '' 2020-08-24T00:59:16.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: continue 2020-08-24T00:59:16.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:59:16.992 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:59:17.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=167503724547 2020-08-24T00:59:17.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 167503724547 2020-08-24T00:59:17.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-163208757251 2-167503724547' 2020-08-24T00:59:17.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:59:17.160 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-163208757251 2020-08-24T00:59:17.160 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:59:17.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:59:17.161 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-163208757251 2020-08-24T00:59:17.161 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:59:17.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=163208757251 2020-08-24T00:59:17.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 163208757251' 2020-08-24T00:59:17.164 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:59:17.164 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 163208757251 2020-08-24T00:59:17.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 0 -lt 163208757251 2020-08-24T00:59:17.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:59:18.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:59:18.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:59:18.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 163208757252 -lt 163208757251 2020-08-24T00:59:18.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:59:18.925 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-167503724547 2020-08-24T00:59:18.925 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:59:18.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:59:18.928 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-167503724547 2020-08-24T00:59:18.929 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:59:18.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=167503724547 2020-08-24T00:59:18.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 167503724547' 2020-08-24T00:59:18.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:59:18.932 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 167503724547 2020-08-24T00:59:19.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 167503724547 -lt 167503724547 2020-08-24T00:59:19.304 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:59:19.304 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:19.304 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:19.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-08-24T00:59:19.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:19.781 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:19.781 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:19.781 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:19.781 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:19.782 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:19.783 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:20.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:20.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:20.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:20.137 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:20.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:20.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T00:59:20.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T00:59:20.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T00:59:20.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:20.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T00:59:20.712 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:20.712 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:20.713 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:20.713 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:20.714 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:20.714 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:20.715 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:20.715 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:21.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:21.091 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:21.091 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:21.091 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:21.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:21.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:21.542 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:21.543 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:21.986 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:21.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:21.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-08-24T00:59:21.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:21.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-08-24T00:59:22.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:22.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:22.184 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:22.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:22.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:22.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:22.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:22.186 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:22.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:22.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:22.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:22.567 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:23.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:23.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:23.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:23.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:23.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:23.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:23.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:23.022 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:23.023 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:23.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:23.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:23.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-08-24T00:59:23.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:23.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-08-24T00:59:23.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:23.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:23.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:23.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:23.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:23.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:23.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:23.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:24.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:24.258 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:24.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:24.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:24.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:24.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:24.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:24.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:24.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:24.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:24.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:24.731 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:24.731 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:25.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:25.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:25.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-08-24T00:59:25.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:25.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-08-24T00:59:25.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:25.986 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:25.986 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:25.987 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:25.987 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:25.987 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:25.988 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:25.988 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:26.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:26.723 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:26.724 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:26.724 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:26.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:26.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:26.840 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:26.840 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:26.840 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:26.841 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:26.841 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:26.841 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:26.842 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:27.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:27.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:27.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-08-24T00:59:27.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:27.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-08-24T00:59:28.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:28.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:28.897 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:28.897 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:28.897 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:28.897 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:28.898 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:28.898 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:29.282 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:29.283 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:29.283 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:29.284 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:29.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:29.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:29.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:29.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:29.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:29.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:29.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:29.764 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:29.764 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:30.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:30.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:30.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-08-24T00:59:30.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:30.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-08-24T00:59:33.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:33.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:33.427 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:33.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:33.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:33.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:33.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:33.430 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:33.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:33.814 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:33.814 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:33.815 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:34.285 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:34.286 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:34.286 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:34.287 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:34.287 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:34.287 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:34.287 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:34.288 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:34.288 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:34.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:34.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:34.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-08-24T00:59:34.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:34.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-08-24T00:59:41.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:41.138 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:41.139 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:41.139 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:41.139 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:41.140 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:41.140 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:41.140 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:41.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:41.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:41.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:41.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:41.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:41.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:41.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:41.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:41.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:41.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:41.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:41.978 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:41.979 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:42.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:42.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:42.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-08-24T00:59:42.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:42.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-08-24T00:59:49.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:49.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:49.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:49.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:49.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:49.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:49.738 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:49.738 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:50.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:50.120 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:50.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:50.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:50.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:50.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:50.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:50.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:50.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:50.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:50.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:50.600 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:50.600 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:51.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:51.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:51.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-08-24T00:59:51.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-08-24T00:59:51.462 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T00:59:51.462 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_fingerprint": "b3fd88c7-150a-4869-93c3-c80ae414149a", 2020-08-24T00:59:51.462 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "15.2.4-661-g0a941e5", 2020-08-24T00:59:51.463 INFO:tasks.workunit.client.0.smithi060.stdout: "commit": "0a941e55043ccee6d35a8717bec55ff7581e67d8", 2020-08-24T00:59:51.463 INFO:tasks.workunit.client.0.smithi060.stdout: "timestamp": "2020-08-24T00:59:51.460184+0000", 2020-08-24T00:59:51.463 INFO:tasks.workunit.client.0.smithi060.stdout: "tag": "", 2020-08-24T00:59:51.463 INFO:tasks.workunit.client.0.smithi060.stdout: "health": { 2020-08-24T00:59:51.463 INFO:tasks.workunit.client.0.smithi060.stdout: "status": "HEALTH_WARN", 2020-08-24T00:59:51.464 INFO:tasks.workunit.client.0.smithi060.stdout: "checks": { 2020-08-24T00:59:51.464 INFO:tasks.workunit.client.0.smithi060.stdout: "OSDMAP_FLAGS": { 2020-08-24T00:59:51.464 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:59:51.464 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:59:51.465 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "nodown,noout,noin flag(s) set", 2020-08-24T00:59:51.465 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 17 2020-08-24T00:59:51.465 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.465 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [], 2020-08-24T00:59:51.466 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:59:51.467 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.467 INFO:tasks.workunit.client.0.smithi060.stdout: "OSD_DOWN": { 2020-08-24T00:59:51.467 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:59:51.467 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:59:51.468 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 osds down", 2020-08-24T00:59:51.468 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:59:51.469 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.469 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:59:51.469 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.469 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "osd.1 (root=default,host=smithi060) is down" 2020-08-24T00:59:51.470 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.470 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.470 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:59:51.470 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.471 INFO:tasks.workunit.client.0.smithi060.stdout: "PG_DEGRADED": { 2020-08-24T00:59:51.471 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:59:51.471 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:59:51.471 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-08-24T00:59:51.471 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 32 2020-08-24T00:59:51.472 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.472 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:59:51.472 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.472 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.472 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.473 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.473 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.473 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.473 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.473 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.475 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.475 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.475 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.475 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.476 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.476 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.476 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.476 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.476 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.477 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.477 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.477 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.477 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.477 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.478 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.478 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.478 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.478 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.478 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.478 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.479 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.479 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.479 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.479 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.480 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.480 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.480 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.480 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.481 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.481 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.481 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.481 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.481 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.482 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.482 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.482 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.482 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.482 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.483 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.483 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.483 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.483 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.484 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.484 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.484 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.484 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.484 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.485 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.485 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.485 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.485 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.485 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.486 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.486 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.486 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.490 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.490 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.490 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.491 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.491 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.491 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.491 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.491 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.492 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.492 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.492 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.492 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.492 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.493 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.493 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.493 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.493 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.493 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.494 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.494 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.494 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.494 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.494 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.495 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.495 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.495 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.495 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.495 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.495 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.496 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.496 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.496 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-08-24T00:59:51.496 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.496 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.497 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:59:51.497 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.497 INFO:tasks.workunit.client.0.smithi060.stdout: "POOL_APP_NOT_ENABLED": { 2020-08-24T00:59:51.497 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T00:59:51.497 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T00:59:51.498 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 pool(s) do not have an application enabled", 2020-08-24T00:59:51.498 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T00:59:51.498 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.498 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T00:59:51.498 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.499 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "application not enabled on pool 'test'" 2020-08-24T00:59:51.499 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.499 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.500 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-08-24T00:59:51.500 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.500 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.500 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T00:59:51.500 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.501 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.501 INFO:tasks.workunit.client.0.smithi060.stdout: "mutes": [] 2020-08-24T00:59:51.501 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.501 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_first_committed": 1, 2020-08-24T00:59:51.501 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_last_committed": 1, 2020-08-24T00:59:51.501 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap": { 2020-08-24T00:59:51.502 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:59:51.502 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:59:51.502 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:58:00.856785Z", 2020-08-24T00:59:51.502 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:58:00.856785Z", 2020-08-24T00:59:51.502 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release": 15, 2020-08-24T00:59:51.503 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release_name": "octopus", 2020-08-24T00:59:51.503 INFO:tasks.workunit.client.0.smithi060.stdout: "features": { 2020-08-24T00:59:51.503 INFO:tasks.workunit.client.0.smithi060.stdout: "persistent": [ 2020-08-24T00:59:51.503 INFO:tasks.workunit.client.0.smithi060.stdout: "kraken", 2020-08-24T00:59:51.503 INFO:tasks.workunit.client.0.smithi060.stdout: "luminous", 2020-08-24T00:59:51.504 INFO:tasks.workunit.client.0.smithi060.stdout: "mimic", 2020-08-24T00:59:51.504 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap-prune", 2020-08-24T00:59:51.504 INFO:tasks.workunit.client.0.smithi060.stdout: "nautilus", 2020-08-24T00:59:51.504 INFO:tasks.workunit.client.0.smithi060.stdout: "octopus" 2020-08-24T00:59:51.504 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.505 INFO:tasks.workunit.client.0.smithi060.stdout: "optional": [] 2020-08-24T00:59:51.505 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.505 INFO:tasks.workunit.client.0.smithi060.stdout: "mons": [ 2020-08-24T00:59:51.506 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.506 INFO:tasks.workunit.client.0.smithi060.stdout: "rank": 0, 2020-08-24T00:59:51.506 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "a", 2020-08-24T00:59:51.507 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:59:51.507 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.507 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.507 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.507 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115", 2020-08-24T00:59:51.507 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 0 2020-08-24T00:59:51.508 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.508 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.508 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.508 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115/0", 2020-08-24T00:59:51.508 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:7115/0", 2020-08-24T00:59:51.509 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 0, 2020-08-24T00:59:51.509 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 0 2020-08-24T00:59:51.509 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.510 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.510 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.510 INFO:tasks.workunit.client.0.smithi060.stdout: "quorum": [ 2020-08-24T00:59:51.510 INFO:tasks.workunit.client.0.smithi060.stdout: 0 2020-08-24T00:59:51.511 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.511 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap": { 2020-08-24T00:59:51.511 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 43, 2020-08-24T00:59:51.511 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T00:59:51.511 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:58:01.596369+0000", 2020-08-24T00:59:51.511 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:59:16.194710+0000", 2020-08-24T00:59:51.512 INFO:tasks.workunit.client.0.smithi060.stdout: "last_up_change": "2020-08-24T00:59:14.574570+0000", 2020-08-24T00:59:51.512 INFO:tasks.workunit.client.0.smithi060.stdout: "last_in_change": "2020-08-24T00:59:15.911184+0000", 2020-08-24T00:59:51.512 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-08-24T00:59:51.512 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_num": 5800384, 2020-08-24T00:59:51.512 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_set": [ 2020-08-24T00:59:51.513 INFO:tasks.workunit.client.0.smithi060.stdout: "nodown", 2020-08-24T00:59:51.513 INFO:tasks.workunit.client.0.smithi060.stdout: "noin", 2020-08-24T00:59:51.513 INFO:tasks.workunit.client.0.smithi060.stdout: "noout", 2020-08-24T00:59:51.513 INFO:tasks.workunit.client.0.smithi060.stdout: "pglog_hardlimit", 2020-08-24T00:59:51.513 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snapdirs", 2020-08-24T00:59:51.514 INFO:tasks.workunit.client.0.smithi060.stdout: "recovery_deletes", 2020-08-24T00:59:51.514 INFO:tasks.workunit.client.0.smithi060.stdout: "sortbitwise" 2020-08-24T00:59:51.514 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.514 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_version": 7, 2020-08-24T00:59:51.515 INFO:tasks.workunit.client.0.smithi060.stdout: "full_ratio": 0.99000000953674316, 2020-08-24T00:59:51.515 INFO:tasks.workunit.client.0.smithi060.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-08-24T00:59:51.515 INFO:tasks.workunit.client.0.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2020-08-24T00:59:51.516 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_snapshot": "", 2020-08-24T00:59:51.516 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_max": 1, 2020-08-24T00:59:51.516 INFO:tasks.workunit.client.0.smithi060.stdout: "max_osd": 3, 2020-08-24T00:59:51.517 INFO:tasks.workunit.client.0.smithi060.stdout: "require_min_compat_client": "jewel", 2020-08-24T00:59:51.517 INFO:tasks.workunit.client.0.smithi060.stdout: "min_compat_client": "jewel", 2020-08-24T00:59:51.517 INFO:tasks.workunit.client.0.smithi060.stdout: "require_osd_release": "octopus", 2020-08-24T00:59:51.517 INFO:tasks.workunit.client.0.smithi060.stdout: "pools": [ 2020-08-24T00:59:51.517 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:59:51.518 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_name": "test", 2020-08-24T00:59:51.518 INFO:tasks.workunit.client.0.smithi060.stdout: "create_time": "2020-08-24T00:58:24.119074+0000", 2020-08-24T00:59:51.518 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": 1, 2020-08-24T00:59:51.518 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_names": "hashpspool", 2020-08-24T00:59:51.519 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:59:51.519 INFO:tasks.workunit.client.0.smithi060.stdout: "size": 3, 2020-08-24T00:59:51.519 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 2, 2020-08-24T00:59:51.519 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_rule": 0, 2020-08-24T00:59:51.519 INFO:tasks.workunit.client.0.smithi060.stdout: "object_hash": 2, 2020-08-24T00:59:51.520 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_autoscale_mode": "on", 2020-08-24T00:59:51.520 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num": 32, 2020-08-24T00:59:51.520 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num": 1, 2020-08-24T00:59:51.520 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num_target": 32, 2020-08-24T00:59:51.520 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_target": 32, 2020-08-24T00:59:51.521 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_pending": 32, 2020-08-24T00:59:51.521 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pg_merge_meta": { 2020-08-24T00:59:51.521 INFO:tasks.workunit.client.0.smithi060.stdout: "source_pgid": "0.0", 2020-08-24T00:59:51.521 INFO:tasks.workunit.client.0.smithi060.stdout: "ready_epoch": 0, 2020-08-24T00:59:51.521 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_started": 0, 2020-08-24T00:59:51.522 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": 0, 2020-08-24T00:59:51.522 INFO:tasks.workunit.client.0.smithi060.stdout: "source_version": "0'0", 2020-08-24T00:59:51.522 INFO:tasks.workunit.client.0.smithi060.stdout: "target_version": "0'0" 2020-08-24T00:59:51.522 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.522 INFO:tasks.workunit.client.0.smithi060.stdout: "last_change": "32", 2020-08-24T00:59:51.523 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend": "0", 2020-08-24T00:59:51.523 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2020-08-24T00:59:51.523 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_preluminous": "32", 2020-08-24T00:59:51.523 INFO:tasks.workunit.client.0.smithi060.stdout: "auid": 0, 2020-08-24T00:59:51.523 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_mode": "selfmanaged", 2020-08-24T00:59:51.524 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_seq": 0, 2020-08-24T00:59:51.524 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_epoch": 0, 2020-08-24T00:59:51.524 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_snaps": [], 2020-08-24T00:59:51.524 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps": "[]", 2020-08-24T00:59:51.524 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_bytes": 0, 2020-08-24T00:59:51.525 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_objects": 0, 2020-08-24T00:59:51.525 INFO:tasks.workunit.client.0.smithi060.stdout: "tiers": [], 2020-08-24T00:59:51.525 INFO:tasks.workunit.client.0.smithi060.stdout: "tier_of": -1, 2020-08-24T00:59:51.525 INFO:tasks.workunit.client.0.smithi060.stdout: "read_tier": -1, 2020-08-24T00:59:51.525 INFO:tasks.workunit.client.0.smithi060.stdout: "write_tier": -1, 2020-08-24T00:59:51.526 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_mode": "none", 2020-08-24T00:59:51.526 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_bytes": 0, 2020-08-24T00:59:51.526 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_objects": 0, 2020-08-24T00:59:51.526 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-08-24T00:59:51.526 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-08-24T00:59:51.527 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2020-08-24T00:59:51.527 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_flush_age": 0, 2020-08-24T00:59:51.527 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_evict_age": 0, 2020-08-24T00:59:51.528 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profile": "", 2020-08-24T00:59:51.528 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_params": { 2020-08-24T00:59:51.529 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "none" 2020-08-24T00:59:51.529 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.529 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_period": 0, 2020-08-24T00:59:51.529 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_count": 0, 2020-08-24T00:59:51.529 INFO:tasks.workunit.client.0.smithi060.stdout: "use_gmt_hitset": true, 2020-08-24T00:59:51.530 INFO:tasks.workunit.client.0.smithi060.stdout: "min_read_recency_for_promote": 0, 2020-08-24T00:59:51.530 INFO:tasks.workunit.client.0.smithi060.stdout: "min_write_recency_for_promote": 0, 2020-08-24T00:59:51.530 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2020-08-24T00:59:51.530 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_search_last_n": 0, 2020-08-24T00:59:51.530 INFO:tasks.workunit.client.0.smithi060.stdout: "grade_table": [], 2020-08-24T00:59:51.531 INFO:tasks.workunit.client.0.smithi060.stdout: "stripe_width": 0, 2020-08-24T00:59:51.531 INFO:tasks.workunit.client.0.smithi060.stdout: "expected_num_objects": 0, 2020-08-24T00:59:51.531 INFO:tasks.workunit.client.0.smithi060.stdout: "fast_read": false, 2020-08-24T00:59:51.531 INFO:tasks.workunit.client.0.smithi060.stdout: "options": {}, 2020-08-24T00:59:51.531 INFO:tasks.workunit.client.0.smithi060.stdout: "application_metadata": {} 2020-08-24T00:59:51.531 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.532 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.532 INFO:tasks.workunit.client.0.smithi060.stdout: "osds": [ 2020-08-24T00:59:51.532 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.532 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:59:51.532 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "765fa77d-4249-474d-a0a3-ac21e9faf855", 2020-08-24T00:59:51.533 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:59:51.533 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:59:51.533 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:59:51.533 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:59:51.533 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 5, 2020-08-24T00:59:51.534 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 30, 2020-08-24T00:59:51.534 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 38, 2020-08-24T00:59:51.534 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 42, 2020-08-24T00:59:51.534 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 33, 2020-08-24T00:59:51.534 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:59:51.534 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:59:51.535 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.535 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.535 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.535 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6802", 2020-08-24T00:59:51.535 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.536 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.536 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.536 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.536 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6803", 2020-08-24T00:59:51.536 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.537 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.537 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.537 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.537 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:59:51.537 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.538 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.538 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.538 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6804", 2020-08-24T00:59:51.538 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.538 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.539 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.539 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.539 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6805", 2020-08-24T00:59:51.539 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.539 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.540 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.540 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.540 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:59:51.540 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.540 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.541 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.541 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6808", 2020-08-24T00:59:51.541 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.541 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.541 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.542 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.542 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6809", 2020-08-24T00:59:51.542 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.542 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.542 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.543 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.543 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:59:51.543 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.543 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.543 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.543 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6806", 2020-08-24T00:59:51.544 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.544 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.544 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.544 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.544 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6807", 2020-08-24T00:59:51.545 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T00:59:51.545 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.545 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.545 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.545 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6803/33546", 2020-08-24T00:59:51.545 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6805/33546", 2020-08-24T00:59:51.546 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6809/33546", 2020-08-24T00:59:51.546 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6807/33546", 2020-08-24T00:59:51.546 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:59:51.546 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:59:51.546 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:59:51.547 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.547 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.547 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.547 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:59:51.547 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "b9f5bd36-904e-4fe9-8925-445d22957285", 2020-08-24T00:59:51.548 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 0, 2020-08-24T00:59:51.548 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:59:51.548 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:59:51.548 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:59:51.548 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 0, 2020-08-24T00:59:51.549 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 0, 2020-08-24T00:59:51.549 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 10, 2020-08-24T00:59:51.549 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 22, 2020-08-24T00:59:51.549 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 34, 2020-08-24T00:59:51.549 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:59:51.550 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:59:51.550 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.550 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.550 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.551 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:59:51.551 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.551 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.551 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.551 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.552 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:59:51.552 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.552 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.552 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.552 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.552 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:59:51.553 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.553 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.553 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.553 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:59:51.553 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.554 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.554 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.554 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.554 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:59:51.554 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.554 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.555 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.555 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.555 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:59:51.555 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.555 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.556 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.556 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:59:51.556 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.556 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.556 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.557 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.557 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:59:51.557 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.557 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.557 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.558 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.558 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:59:51.558 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.558 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.558 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.559 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:59:51.559 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.559 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.559 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.559 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.559 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:59:51.560 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T00:59:51.560 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.560 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.560 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.560 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/29117", 2020-08-24T00:59:51.561 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/29117", 2020-08-24T00:59:51.561 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/29117", 2020-08-24T00:59:51.561 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/29117", 2020-08-24T00:59:51.562 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:59:51.562 INFO:tasks.workunit.client.0.smithi060.stdout: "exists" 2020-08-24T00:59:51.562 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.562 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.562 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.563 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:59:51.563 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "03903bbb-6dd8-4f53-b407-7ee8ec7022ce", 2020-08-24T00:59:51.563 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T00:59:51.563 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T00:59:51.563 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T00:59:51.564 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T00:59:51.564 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 15, 2020-08-24T00:59:51.564 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 30, 2020-08-24T00:59:51.564 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 39, 2020-08-24T00:59:51.564 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 0, 2020-08-24T00:59:51.564 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 35, 2020-08-24T00:59:51.565 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T00:59:51.565 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T00:59:51.565 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.565 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.565 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.566 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T00:59:51.566 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.566 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.566 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.566 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.567 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T00:59:51.567 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.567 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.567 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.567 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.567 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T00:59:51.568 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.568 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.568 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.568 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T00:59:51.568 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.569 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.569 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.569 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.569 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T00:59:51.569 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.570 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.570 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.570 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.570 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T00:59:51.570 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.570 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.571 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.571 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T00:59:51.571 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.571 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.571 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.572 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.572 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T00:59:51.572 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.572 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.572 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.572 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.573 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T00:59:51.573 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T00:59:51.573 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.573 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T00:59:51.573 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T00:59:51.574 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.574 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.574 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.574 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T00:59:51.574 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T00:59:51.574 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T00:59:51.575 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.575 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.575 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.575 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/33740", 2020-08-24T00:59:51.575 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/33740", 2020-08-24T00:59:51.576 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/33740", 2020-08-24T00:59:51.576 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/33740", 2020-08-24T00:59:51.576 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T00:59:51.576 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T00:59:51.576 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T00:59:51.577 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.577 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.577 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.577 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_xinfo": [ 2020-08-24T00:59:51.577 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.578 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:59:51.578 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:59:08.073077+0000", 2020-08-24T00:59:51.578 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:59:51.578 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:59:51.578 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:59:51.578 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:59:51.579 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:58:08.576478+0000", 2020-08-24T00:59:51.579 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:59:51.579 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.579 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.579 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:59:51.580 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:59:08.621845+0000", 2020-08-24T00:59:51.580 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:59:51.580 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:59:51.580 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:59:51.580 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:59:51.581 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:58:15.105595+0000", 2020-08-24T00:59:51.581 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:59:51.581 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.581 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.581 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:59:51.581 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:59:09.139471+0000", 2020-08-24T00:59:51.582 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T00:59:51.582 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T00:59:51.582 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T00:59:51.582 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T00:59:51.582 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:58:21.928359+0000", 2020-08-24T00:59:51.583 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T00:59:51.583 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.583 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.583 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap": [], 2020-08-24T00:59:51.583 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap_items": [], 2020-08-24T00:59:51.583 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_temp": [], 2020-08-24T00:59:51.584 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_temp": [], 2020-08-24T00:59:51.584 INFO:tasks.workunit.client.0.smithi060.stdout: "blacklist": {}, 2020-08-24T00:59:51.584 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profiles": { 2020-08-24T00:59:51.584 INFO:tasks.workunit.client.0.smithi060.stdout: "default": { 2020-08-24T00:59:51.584 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T00:59:51.585 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "2", 2020-08-24T00:59:51.585 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T00:59:51.585 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van" 2020-08-24T00:59:51.585 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.585 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.585 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps_queue": [], 2020-08-24T00:59:51.586 INFO:tasks.workunit.client.0.smithi060.stdout: "new_removed_snaps": [], 2020-08-24T00:59:51.586 INFO:tasks.workunit.client.0.smithi060.stdout: "new_purged_snaps": [], 2020-08-24T00:59:51.586 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_node_flags": {}, 2020-08-24T00:59:51.586 INFO:tasks.workunit.client.0.smithi060.stdout: "device_class_flags": {} 2020-08-24T00:59:51.586 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.587 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_metadata": [ 2020-08-24T00:59:51.587 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.587 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:59:51.587 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:59:51.587 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6804/33546,v1:127.0.0.1:6805/33546]", 2020-08-24T00:59:51.587 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:59:51.588 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:59:51.588 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:59:51.588 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:59:51.588 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:59:51.588 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:59:51.589 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:59:51.589 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:59:51.589 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:59:51.589 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-08-24T00:59:51.589 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:59:51.590 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:59:51.590 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:59:51.590 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:59:51.590 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:59:51.591 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:59:51.591 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:59:51.591 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:59:51.591 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:59:51.591 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:59:51.592 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:59:51.592 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:59:51.593 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:59:51.594 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:59:51.594 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:59:51.594 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6802/33546,v1:127.0.0.1:6803/33546]", 2020-08-24T00:59:51.594 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:59:51.594 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/33546,v1:127.0.0.1:6809/33546]", 2020-08-24T00:59:51.595 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/33546,v1:127.0.0.1:6807/33546]", 2020-08-24T00:59:51.595 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:59:51.595 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:59:51.595 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:59:51.595 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:59:51.596 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:59:51.596 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:59:51.596 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:59:51.596 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:59:51.596 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:59:51.597 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/0", 2020-08-24T00:59:51.597 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:59:51.597 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:59:51.597 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:59:51.597 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.598 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.598 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:59:51.598 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:59:51.598 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/29117,v1:127.0.0.1:6813/29117]", 2020-08-24T00:59:51.598 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:59:51.599 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:59:51.599 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:59:51.599 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:59:51.599 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:59:51.599 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:59:51.599 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:59:51.600 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:59:51.600 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:59:51.600 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-08-24T00:59:51.600 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:59:51.600 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:59:51.601 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:59:51.601 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:59:51.602 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:59:51.602 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:59:51.602 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:59:51.602 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:59:51.602 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:59:51.603 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:59:51.603 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:59:51.603 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:59:51.603 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:59:51.603 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:59:51.604 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:59:51.604 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/29117,v1:127.0.0.1:6811/29117]", 2020-08-24T00:59:51.604 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:59:51.604 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/29117,v1:127.0.0.1:6817/29117]", 2020-08-24T00:59:51.605 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/29117,v1:127.0.0.1:6815/29117]", 2020-08-24T00:59:51.605 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:59:51.605 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:59:51.605 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:59:51.606 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:59:51.606 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:59:51.606 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:59:51.607 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:59:51.607 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:59:51.607 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:59:51.608 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/1", 2020-08-24T00:59:51.608 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:59:51.608 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:59:51.608 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:59:51.608 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.609 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.609 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:59:51.609 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T00:59:51.609 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/33740,v1:127.0.0.1:6813/33740]", 2020-08-24T00:59:51.609 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T00:59:51.617 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T00:59:51.617 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T00:59:51.617 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T00:59:51.617 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T00:59:51.618 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T00:59:51.618 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T00:59:51.618 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T00:59:51.618 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T00:59:51.618 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-08-24T00:59:51.619 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T00:59:51.619 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T00:59:51.619 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T00:59:51.619 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T00:59:51.619 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T00:59:51.620 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T00:59:51.620 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T00:59:51.620 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T00:59:51.620 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T00:59:51.621 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T00:59:51.621 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T00:59:51.621 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T00:59:51.621 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T00:59:51.622 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T00:59:51.622 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T00:59:51.622 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/33740,v1:127.0.0.1:6811/33740]", 2020-08-24T00:59:51.622 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T00:59:51.622 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/33740,v1:127.0.0.1:6817/33740]", 2020-08-24T00:59:51.623 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/33740,v1:127.0.0.1:6815/33740]", 2020-08-24T00:59:51.623 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T00:59:51.623 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T00:59:51.623 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T00:59:51.623 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T00:59:51.624 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T00:59:51.624 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T00:59:51.624 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T00:59:51.624 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T00:59:51.624 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T00:59:51.625 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/2", 2020-08-24T00:59:51.625 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T00:59:51.625 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T00:59:51.626 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T00:59:51.626 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.626 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.627 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_clean_epochs": { 2020-08-24T00:59:51.627 INFO:tasks.workunit.client.0.smithi060.stdout: "min_last_epoch_clean": 0, 2020-08-24T00:59:51.627 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": { 2020-08-24T00:59:51.627 INFO:tasks.workunit.client.0.smithi060.stdout: "per_pool": [] 2020-08-24T00:59:51.628 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.628 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_epochs": [ 2020-08-24T00:59:51.628 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.628 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:59:51.628 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 38 2020-08-24T00:59:51.629 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.629 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.629 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:59:51.629 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 10 2020-08-24T00:59:51.629 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.630 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.630 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:59:51.632 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 39 2020-08-24T00:59:51.632 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.632 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.632 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.633 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_first_committed": 1, 2020-08-24T00:59:51.633 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_last_committed": 43, 2020-08-24T00:59:51.633 INFO:tasks.workunit.client.0.smithi060.stdout: "crushmap": { 2020-08-24T00:59:51.633 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": [ 2020-08-24T00:59:51.633 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.634 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:59:51.634 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.0", 2020-08-24T00:59:51.634 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:59:51.634 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.634 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.635 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:59:51.635 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.1", 2020-08-24T00:59:51.635 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:59:51.635 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.636 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.636 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:59:51.636 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.2", 2020-08-24T00:59:51.636 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T00:59:51.636 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.637 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.637 INFO:tasks.workunit.client.0.smithi060.stdout: "types": [ 2020-08-24T00:59:51.637 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.637 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 0, 2020-08-24T00:59:51.637 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd" 2020-08-24T00:59:51.638 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.638 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.638 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:59:51.638 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "host" 2020-08-24T00:59:51.638 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.639 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.639 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 2, 2020-08-24T00:59:51.639 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "chassis" 2020-08-24T00:59:51.639 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.639 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.639 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 3, 2020-08-24T00:59:51.640 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "rack" 2020-08-24T00:59:51.640 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.640 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.640 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 4, 2020-08-24T00:59:51.640 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "row" 2020-08-24T00:59:51.641 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.641 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.641 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 5, 2020-08-24T00:59:51.641 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pdu" 2020-08-24T00:59:51.641 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.642 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.642 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 6, 2020-08-24T00:59:51.642 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pod" 2020-08-24T00:59:51.642 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.642 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.642 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 7, 2020-08-24T00:59:51.643 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "room" 2020-08-24T00:59:51.643 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.643 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.643 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 8, 2020-08-24T00:59:51.643 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "datacenter" 2020-08-24T00:59:51.644 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.644 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.644 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 9, 2020-08-24T00:59:51.644 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "zone" 2020-08-24T00:59:51.644 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.645 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.645 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 10, 2020-08-24T00:59:51.645 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "region" 2020-08-24T00:59:51.645 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.645 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.645 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:59:51.646 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "root" 2020-08-24T00:59:51.646 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.646 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.646 INFO:tasks.workunit.client.0.smithi060.stdout: "buckets": [ 2020-08-24T00:59:51.646 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.647 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -1, 2020-08-24T00:59:51.647 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default", 2020-08-24T00:59:51.647 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:59:51.647 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:59:51.647 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:59:51.648 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:59:51.648 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:59:51.648 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:59:51.648 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.648 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:59:51.649 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:59:51.649 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:59:51.649 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.649 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.649 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.650 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.650 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -2, 2020-08-24T00:59:51.650 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default~hdd", 2020-08-24T00:59:51.650 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T00:59:51.650 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T00:59:51.651 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:59:51.651 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:59:51.651 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:59:51.651 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:59:51.651 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.652 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:59:51.652 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:59:51.652 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:59:51.652 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.653 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.653 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.653 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.653 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T00:59:51.653 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060", 2020-08-24T00:59:51.654 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:59:51.654 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:59:51.654 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:59:51.654 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:59:51.654 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:59:51.654 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:59:51.655 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.655 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:59:51.655 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:59:51.655 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:59:51.655 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.656 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.656 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:59:51.656 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:59:51.656 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:59:51.656 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.657 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.657 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:59:51.657 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:59:51.657 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:59:51.657 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.658 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.658 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.658 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.658 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T00:59:51.658 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060~hdd", 2020-08-24T00:59:51.659 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T00:59:51.659 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T00:59:51.659 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T00:59:51.659 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T00:59:51.659 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T00:59:51.660 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T00:59:51.660 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.660 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T00:59:51.660 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:59:51.660 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T00:59:51.660 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.661 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.661 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T00:59:51.661 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:59:51.661 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T00:59:51.661 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.662 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.662 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T00:59:51.662 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T00:59:51.662 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T00:59:51.662 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.663 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.663 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.663 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.663 INFO:tasks.workunit.client.0.smithi060.stdout: "rules": [ 2020-08-24T00:59:51.663 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.663 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 0, 2020-08-24T00:59:51.664 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "replicated_rule", 2020-08-24T00:59:51.664 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 0, 2020-08-24T00:59:51.664 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T00:59:51.664 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 1, 2020-08-24T00:59:51.664 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 10, 2020-08-24T00:59:51.665 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T00:59:51.665 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.665 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T00:59:51.665 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T00:59:51.666 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T00:59:51.666 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.666 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.666 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_firstn", 2020-08-24T00:59:51.666 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T00:59:51.667 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T00:59:51.667 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.667 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.667 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T00:59:51.667 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.668 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T00:59:51.668 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.668 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.668 INFO:tasks.workunit.client.0.smithi060.stdout: "tunables": { 2020-08-24T00:59:51.668 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_tries": 0, 2020-08-24T00:59:51.669 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_fallback_tries": 0, 2020-08-24T00:59:51.669 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_total_tries": 50, 2020-08-24T00:59:51.669 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_descend_once": 1, 2020-08-24T00:59:51.669 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_vary_r": 1, 2020-08-24T00:59:51.669 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_stable": 1, 2020-08-24T00:59:51.670 INFO:tasks.workunit.client.0.smithi060.stdout: "straw_calc_version": 1, 2020-08-24T00:59:51.670 INFO:tasks.workunit.client.0.smithi060.stdout: "allowed_bucket_algs": 54, 2020-08-24T00:59:51.670 INFO:tasks.workunit.client.0.smithi060.stdout: "profile": "jewel", 2020-08-24T00:59:51.670 INFO:tasks.workunit.client.0.smithi060.stdout: "optimal_tunables": 1, 2020-08-24T00:59:51.670 INFO:tasks.workunit.client.0.smithi060.stdout: "legacy_tunables": 0, 2020-08-24T00:59:51.671 INFO:tasks.workunit.client.0.smithi060.stdout: "minimum_required_version": "jewel", 2020-08-24T00:59:51.671 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables": 1, 2020-08-24T00:59:51.671 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables2": 1, 2020-08-24T00:59:51.671 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v2_rules": 0, 2020-08-24T00:59:51.671 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables3": 1, 2020-08-24T00:59:51.671 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v3_rules": 0, 2020-08-24T00:59:51.672 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v4_buckets": 1, 2020-08-24T00:59:51.672 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables5": 1, 2020-08-24T00:59:51.672 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v5_rules": 0 2020-08-24T00:59:51.672 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.672 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_args": {} 2020-08-24T00:59:51.673 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.673 INFO:tasks.workunit.client.0.smithi060.stdout: "fsmap": { 2020-08-24T00:59:51.673 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:59:51.673 INFO:tasks.workunit.client.0.smithi060.stdout: "default_fscid": -1, 2020-08-24T00:59:51.673 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": { 2020-08-24T00:59:51.674 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": {}, 2020-08-24T00:59:51.674 INFO:tasks.workunit.client.0.smithi060.stdout: "ro_compat": {}, 2020-08-24T00:59:51.674 INFO:tasks.workunit.client.0.smithi060.stdout: "incompat": { 2020-08-24T00:59:51.674 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_1": "base v0.20", 2020-08-24T00:59:51.674 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_2": "client writeable ranges", 2020-08-24T00:59:51.675 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_3": "default file layouts on dirs", 2020-08-24T00:59:51.675 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_4": "dir inode in separate object", 2020-08-24T00:59:51.675 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_5": "mds uses versioned encoding", 2020-08-24T00:59:51.675 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_6": "dirfrag is stored in omap", 2020-08-24T00:59:51.676 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_8": "no anchor table", 2020-08-24T00:59:51.676 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_9": "file layout v2", 2020-08-24T00:59:51.676 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_10": "snaprealm v2" 2020-08-24T00:59:51.676 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.676 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.676 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_flags": { 2020-08-24T00:59:51.677 INFO:tasks.workunit.client.0.smithi060.stdout: "enable_multiple": false, 2020-08-24T00:59:51.677 INFO:tasks.workunit.client.0.smithi060.stdout: "ever_enabled_multiple": false 2020-08-24T00:59:51.677 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.677 INFO:tasks.workunit.client.0.smithi060.stdout: "standbys": [], 2020-08-24T00:59:51.678 INFO:tasks.workunit.client.0.smithi060.stdout: "filesystems": [] 2020-08-24T00:59:51.678 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.678 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_first_committed": 1, 2020-08-24T00:59:51.678 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_last_committed": 1, 2020-08-24T00:59:51.678 INFO:tasks.workunit.client.0.smithi060.stdout: "auth": { 2020-08-24T00:59:51.679 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:59:51.679 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 5, 2020-08-24T00:59:51.679 INFO:tasks.workunit.client.0.smithi060.stdout: "num_secrets": 3 2020-08-24T00:59:51.679 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.679 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 32, 2020-08-24T00:59:51.680 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_active": 32, 2020-08-24T00:59:51.680 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_unknown": 0, 2020-08-24T00:59:51.680 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osd": 3, 2020-08-24T00:59:51.680 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_sum": { 2020-08-24T00:59:51.680 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:59:51.681 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61700, 2020-08-24T00:59:51.681 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:59:51.681 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:59:51.681 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:59:51.682 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:59:51.682 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:59:51.682 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:59:51.682 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:59:51.682 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:59:51.683 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:59:51.683 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:59:51.683 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:59:51.683 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:59:51.683 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 100, 2020-08-24T00:59:51.684 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 100, 2020-08-24T00:59:51.684 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:59:51.684 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:59:51.684 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:59:51.684 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:59:51.684 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:59:51.685 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:59:51.685 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:59:51.685 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:59:51.685 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:59:51.685 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:59:51.686 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:59:51.686 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:59:51.686 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:59:51.686 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:59:51.686 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:59:51.687 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:59:51.687 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:59:51.687 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:59:51.687 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:59:51.687 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:59:51.688 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:59:51.688 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:59:51.688 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:59:51.688 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:59:51.688 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:59:51.688 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.689 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:59:51.689 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:59:51.689 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:59:51.689 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:59:51.689 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 0, 2020-08-24T00:59:51.690 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 0, 2020-08-24T00:59:51.690 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:59:51.690 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:59:51.690 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:59:51.690 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:59:51.691 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:59:51.691 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.691 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 512, 2020-08-24T00:59:51.691 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 512, 2020-08-24T00:59:51.691 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 64, 2020-08-24T00:59:51.691 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 64, 2020-08-24T00:59:51.692 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 0 2020-08-24T00:59:51.692 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.692 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum": { 2020-08-24T00:59:51.692 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:59:51.692 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:59:51.693 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 65, 2020-08-24T00:59:51.693 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:59:51.693 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:59:51.693 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:59:51.693 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:59:51.694 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3180288, 2020-08-24T00:59:51.694 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 34240, 2020-08-24T00:59:51.694 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 6, 2020-08-24T00:59:51.694 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145721, 2020-08-24T00:59:51.694 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311392512, 2020-08-24T00:59:51.694 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:59:51.695 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:59:51.695 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318865932288, 2020-08-24T00:59:51.695 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:59:51.695 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 35061760, 2020-08-24T00:59:51.695 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 480196, 2020-08-24T00:59:51.696 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:59:51.696 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:59:51.696 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:59:51.696 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 6646, 2020-08-24T00:59:51.696 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221218826 2020-08-24T00:59:51.696 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.697 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:59:51.697 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:59:51.697 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:59:51.697 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T00:59:51.697 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:59:51.698 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:59:51.698 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:59:51.698 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.698 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:59:51.698 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:59:51.699 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:59:51.699 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:59:51.699 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:59:51.699 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.699 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:59:51.699 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:59:51.700 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.700 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum_by_class": { 2020-08-24T00:59:51.700 INFO:tasks.workunit.client.0.smithi060.stdout: "hdd": { 2020-08-24T00:59:51.700 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T00:59:51.700 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T00:59:51.701 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 65, 2020-08-24T00:59:51.701 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T00:59:51.701 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T00:59:51.701 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T00:59:51.701 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T00:59:51.701 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3180288, 2020-08-24T00:59:51.702 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 34240, 2020-08-24T00:59:51.702 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 6, 2020-08-24T00:59:51.702 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145721, 2020-08-24T00:59:51.702 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311392512, 2020-08-24T00:59:51.702 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T00:59:51.703 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T00:59:51.703 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318865932288, 2020-08-24T00:59:51.703 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T00:59:51.703 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 35061760, 2020-08-24T00:59:51.703 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 480196, 2020-08-24T00:59:51.704 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:59:51.704 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:59:51.704 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:59:51.704 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 6646, 2020-08-24T00:59:51.704 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221218826 2020-08-24T00:59:51.704 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.705 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T00:59:51.705 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T00:59:51.705 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T00:59:51.705 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T00:59:51.706 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T00:59:51.706 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T00:59:51.706 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T00:59:51.706 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.706 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T00:59:51.707 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T00:59:51.707 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T00:59:51.707 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T00:59:51.707 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T00:59:51.708 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.708 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T00:59:51.708 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T00:59:51.708 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.708 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.709 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_stats": [ 2020-08-24T00:59:51.709 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.709 INFO:tasks.workunit.client.0.smithi060.stdout: "poolid": 1, 2020-08-24T00:59:51.709 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 32, 2020-08-24T00:59:51.709 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T00:59:51.710 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61700, 2020-08-24T00:59:51.710 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T00:59:51.710 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T00:59:51.710 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T00:59:51.710 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T00:59:51.711 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T00:59:51.711 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T00:59:51.711 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T00:59:51.711 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T00:59:51.711 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T00:59:51.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T00:59:51.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T00:59:51.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T00:59:51.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 100, 2020-08-24T00:59:51.712 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 100, 2020-08-24T00:59:51.713 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T00:59:51.713 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T00:59:51.713 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T00:59:51.713 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T00:59:51.714 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T00:59:51.714 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T00:59:51.714 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T00:59:51.714 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T00:59:51.714 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T00:59:51.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T00:59:51.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T00:59:51.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T00:59:51.715 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T00:59:51.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T00:59:51.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T00:59:51.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T00:59:51.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T00:59:51.716 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T00:59:51.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T00:59:51.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T00:59:51.717 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T00:59:51.718 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T00:59:51.718 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T00:59:51.719 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T00:59:51.719 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T00:59:51.719 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.720 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T00:59:51.720 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T00:59:51.720 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T00:59:51.720 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T00:59:51.721 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 19529728, 2020-08-24T00:59:51.721 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 184174, 2020-08-24T00:59:51.721 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T00:59:51.721 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T00:59:51.721 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T00:59:51.722 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T00:59:51.722 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T00:59:51.722 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.722 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 512, 2020-08-24T00:59:51.723 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 512, 2020-08-24T00:59:51.723 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 64, 2020-08-24T00:59:51.723 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 64, 2020-08-24T00:59:51.723 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 3 2020-08-24T00:59:51.723 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.724 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.724 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_stats": [ 2020-08-24T00:59:51.724 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.724 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:59:51.725 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 163208757258 2020-08-24T00:59:51.725 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.725 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.726 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:59:51.726 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 42949672977 2020-08-24T00:59:51.726 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.726 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.727 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:59:51.727 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 167503724554 2020-08-24T00:59:51.727 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.727 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.727 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_state": [ 2020-08-24T00:59:51.728 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.728 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "active+undersized+degraded", 2020-08-24T00:59:51.728 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 32 2020-08-24T00:59:51.728 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.728 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.729 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_osd": [ 2020-08-24T00:59:51.729 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.729 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T00:59:51.729 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:59:51.730 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 32, 2020-08-24T00:59:51.730 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:59:51.730 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.730 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.731 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T00:59:51.731 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T00:59:51.731 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 0, 2020-08-24T00:59:51.731 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:59:51.732 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.732 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.732 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T00:59:51.732 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 32, 2020-08-24T00:59:51.733 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 32, 2020-08-24T00:59:51.733 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T00:59:51.733 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.733 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.733 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": [ 2020-08-24T00:59:51.734 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T00:59:51.734 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T00:59:51.734 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": {} 2020-08-24T00:59:51.734 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.734 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T00:59:51.735 INFO:tasks.workunit.client.0.smithi060.stdout: "servicemap": { 2020-08-24T00:59:51.735 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T00:59:51.735 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:58:01.597183+0000", 2020-08-24T00:59:51.735 INFO:tasks.workunit.client.0.smithi060.stdout: "services": {} 2020-08-24T00:59:51.735 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T00:59:51.736 INFO:tasks.workunit.client.0.smithi060.stdout: "paxos": { 2020-08-24T00:59:51.736 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T00:59:51.736 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 359, 2020-08-24T00:59:51.736 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pn": 0, 2020-08-24T00:59:51.736 INFO:tasks.workunit.client.0.smithi060.stdout: "accepted_pn": 0 2020-08-24T00:59:51.737 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T00:59:51.737 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T00:59:51.738 INFO:tasks.workunit.client.0.smithi060.stderr:report 99230172 2020-08-24T00:59:51.738 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-08-24T00:59:51.739 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-08-24T00:59:51.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-08-24T00:59:51.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-08-24T00:59:51.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-08-24T00:59:51.739 INFO:tasks.workunit.client.0.smithi060.stdout:writing non-divergent object existing_4 2020-08-24T00:59:51.804 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T00:59:51.804 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596654+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.804 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595585+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.805 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596617+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.805 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.598040+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.805 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595917+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.805 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498491+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.805 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498693+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.806 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595671+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.806 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597434+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.806 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595309+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.807 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594225+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.807 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593998+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.807 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595105+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.807 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596904+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.807 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594689+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.808 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594715+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.808 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594497+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.808 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594556+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.808 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595667+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.808 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594190+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.809 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596910+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.809 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595302+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.809 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596159+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.810 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498563+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.810 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595022+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.811 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593526+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.811 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597416+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.811 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593709+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.811 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593683+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.812 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596397+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.812 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594050+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.812 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595020+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T00:59:51.812 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T00:59:51.813 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T00:59:51.813 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T00:59:51.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-08-24T00:59:51.905 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T00:59:51.905 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-08-24T00:59:51.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T00:59:51.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T00:59:51.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T00:59:51.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T00:59:51.907 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T00:59:51.907 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T00:59:51.908 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T00:59:51.908 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T00:59:51.908 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T00:59:51.908 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T00:59:52.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T00:59:52.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T00:59:52.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T00:59:52.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T00:59:52.035 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T00:59:52.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T00:59:52.405 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T00:59:52.405 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T00:59:52.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T00:59:52.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:59:52.406 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T00:59:52.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=163208757261 2020-08-24T00:59:52.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 163208757261 2020-08-24T00:59:52.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-163208757261' 2020-08-24T00:59:52.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:59:52.568 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T00:59:52.687 INFO:tasks.workunit.client.0.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-08-24T00:59:52.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-08-24T00:59:52.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z '' 2020-08-24T00:59:52.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: continue 2020-08-24T00:59:52.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T00:59:52.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T00:59:52.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=167503724556 2020-08-24T00:59:52.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 167503724556 2020-08-24T00:59:52.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-163208757261 2-167503724556' 2020-08-24T00:59:52.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:59:52.861 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-163208757261 2020-08-24T00:59:52.861 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:59:52.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T00:59:52.862 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-163208757261 2020-08-24T00:59:52.862 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:59:52.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=163208757261 2020-08-24T00:59:52.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 163208757261' 2020-08-24T00:59:52.864 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 163208757261 2020-08-24T00:59:52.865 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:59:53.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 163208757258 -lt 163208757261 2020-08-24T00:59:53.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T00:59:54.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T00:59:54.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T00:59:54.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 163208757261 -lt 163208757261 2020-08-24T00:59:54.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T00:59:54.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-167503724556 2020-08-24T00:59:54.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T00:59:54.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T00:59:54.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-167503724556 2020-08-24T00:59:54.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T00:59:54.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=167503724556 2020-08-24T00:59:54.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 167503724556' 2020-08-24T00:59:54.624 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T00:59:54.624 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 167503724556 2020-08-24T00:59:54.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 167503724556 -lt 167503724556 2020-08-24T00:59:54.993 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T00:59:54.993 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:54.993 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:55.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-08-24T00:59:55.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:55.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:55.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:55.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:55.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:55.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:55.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:55.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:55.827 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:55.827 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:55.828 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:56.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:56.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T00:59:56.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T00:59:56.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T00:59:56.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:56.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T00:59:56.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:56.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:56.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:56.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:56.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:56.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:56.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:56.379 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:56.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:56.734 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:56.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:56.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:57.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:57.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:57.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:57.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:57.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:57.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:57.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:57.192 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:57.192 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:57.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:57.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:57.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-08-24T00:59:57.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:57.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-08-24T00:59:57.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:58.210 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:58.211 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:58.211 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:58.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:58.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:58.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:58.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:58.281 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:58.281 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:58.281 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:58.282 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T00:59:58.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T00:59:58.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T00:59:58.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T00:59:58.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T00:59:58.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T00:59:58.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T00:59:58.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T00:59:58.773 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T00:59:58.773 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T00:59:59.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T00:59:59.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T00:59:59.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-08-24T00:59:59.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T00:59:59.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-08-24T00:59:59.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T00:59:59.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T00:59:59.609 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T00:59:59.609 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T00:59:59.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T00:59:59.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T00:59:59.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T00:59:59.611 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T00:59:59.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T00:59:59.978 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T00:59:59.979 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T00:59:59.979 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:00:00.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T01:00:00.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:00:00.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:00:00.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:00:00.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:00:00.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:00:00.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:00:00.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:00:00.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:00:00.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:00:00.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:00:00.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-08-24T01:00:00.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:00:00.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-08-24T01:00:01.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:00:01.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:00:01.709 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:00:01.709 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:00:01.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:00:01.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:00:01.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:00:01.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:00:02.095 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:00:02.096 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:00:02.097 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:00:02.097 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:00:02.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T01:00:02.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:00:02.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:00:02.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:00:02.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:00:02.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:00:02.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:00:02.556 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:00:02.557 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:00:03.003 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:00:03.003 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:00:03.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-08-24T01:00:03.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:00:03.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-08-24T01:00:04.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:00:04.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:00:04.605 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:00:04.605 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:00:04.605 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:00:04.605 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:00:04.606 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:00:04.606 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:00:04.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:00:04.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:00:04.991 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:00:04.991 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:00:05.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T01:00:05.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:00:05.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:00:05.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:00:05.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:00:05.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:00:05.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:00:05.470 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:00:05.470 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:00:05.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:00:05.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:00:05.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-08-24T01:00:05.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:00:05.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-08-24T01:00:09.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:00:09.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:00:09.130 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:00:09.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:00:09.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:00:09.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:00:09.132 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:00:09.132 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:00:09.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:00:09.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:00:09.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:00:09.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:00:09.964 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T01:00:09.964 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:00:09.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:00:09.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:00:09.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:00:09.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:00:09.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:00:09.966 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:00:09.966 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:00:10.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:00:10.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:00:10.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-08-24T01:00:10.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:00:10.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-08-24T01:00:16.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:00:16.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:00:16.811 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:00:16.811 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:00:16.812 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:00:16.813 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:00:16.813 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:00:16.813 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:00:17.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:00:17.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:00:17.186 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:00:17.186 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:00:17.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T01:00:17.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:00:17.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:00:17.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:00:17.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:00:17.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:00:17.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:00:17.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:00:17.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:00:18.125 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:00:18.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:00:18.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-08-24T01:00:18.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:00:18.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-08-24T01:00:25.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:00:25.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:00:25.434 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:00:25.434 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:00:25.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:00:25.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:00:25.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:00:25.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:00:25.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:00:25.819 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:00:25.819 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:00:25.820 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:00:26.272 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-08-24T01:00:26.272 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:00:26.272 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:00:26.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:00:26.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:00:26.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:00:26.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:00:26.274 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:00:26.274 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:00:26.732 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:00:26.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:00:26.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-08-24T01:00:26.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-08-24T01:00:27.189 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:00:27.190 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_fingerprint": "b3fd88c7-150a-4869-93c3-c80ae414149a", 2020-08-24T01:00:27.190 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "15.2.4-661-g0a941e5", 2020-08-24T01:00:27.190 INFO:tasks.workunit.client.0.smithi060.stdout: "commit": "0a941e55043ccee6d35a8717bec55ff7581e67d8", 2020-08-24T01:00:27.190 INFO:tasks.workunit.client.0.smithi060.stdout: "timestamp": "2020-08-24T01:00:27.185955+0000", 2020-08-24T01:00:27.190 INFO:tasks.workunit.client.0.smithi060.stdout: "tag": "", 2020-08-24T01:00:27.191 INFO:tasks.workunit.client.0.smithi060.stdout: "health": { 2020-08-24T01:00:27.191 INFO:tasks.workunit.client.0.smithi060.stdout: "status": "HEALTH_WARN", 2020-08-24T01:00:27.191 INFO:tasks.workunit.client.0.smithi060.stdout: "checks": { 2020-08-24T01:00:27.191 INFO:tasks.workunit.client.0.smithi060.stdout: "OSDMAP_FLAGS": { 2020-08-24T01:00:27.191 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:00:27.192 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:00:27.192 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "nodown,noout,noin flag(s) set", 2020-08-24T01:00:27.192 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 17 2020-08-24T01:00:27.192 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.192 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [], 2020-08-24T01:00:27.193 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:00:27.193 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.193 INFO:tasks.workunit.client.0.smithi060.stdout: "OSD_DOWN": { 2020-08-24T01:00:27.193 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:00:27.193 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:00:27.194 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 osds down", 2020-08-24T01:00:27.194 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:00:27.194 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.196 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:00:27.196 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.196 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "osd.1 (root=default,host=smithi060) is down" 2020-08-24T01:00:27.196 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.197 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.197 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:00:27.197 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.197 INFO:tasks.workunit.client.0.smithi060.stdout: "PG_DEGRADED": { 2020-08-24T01:00:27.197 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:00:27.198 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:00:27.198 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-08-24T01:00:27.198 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 64 2020-08-24T01:00:27.198 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.198 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:00:27.199 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.199 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.199 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.199 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.199 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.200 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.200 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.200 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.2 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.200 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.200 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.201 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.3 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.201 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.201 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.202 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.4 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.202 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.203 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.203 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.5 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.203 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.203 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.203 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.6 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.204 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.204 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.204 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.7 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.204 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.204 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.205 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.8 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.205 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.205 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.205 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.9 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.206 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.206 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.206 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.a is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.207 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.207 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.207 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.b is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.207 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.207 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.208 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.c is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.208 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.208 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.208 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.d is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.208 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.209 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.209 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.209 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.209 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.210 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.210 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.211 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.211 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.10 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.211 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.211 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.211 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.11 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.212 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.212 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.212 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.12 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.212 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.212 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.213 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.13 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.213 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.213 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.213 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.14 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.213 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.214 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.214 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.15 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.214 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.214 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.214 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.16 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.215 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.215 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.215 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.17 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.215 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.215 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.216 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.18 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.216 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.216 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.216 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.19 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.216 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.217 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.217 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1a is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.217 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.217 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.217 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1b is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.218 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.218 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.218 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1c is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.227 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.227 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.228 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1d is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.228 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.228 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.228 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1e is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.228 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.229 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.229 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.1f is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-08-24T01:00:27.229 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.229 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.229 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:00:27.230 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.230 INFO:tasks.workunit.client.0.smithi060.stdout: "POOL_APP_NOT_ENABLED": { 2020-08-24T01:00:27.230 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:00:27.231 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:00:27.231 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 pool(s) do not have an application enabled", 2020-08-24T01:00:27.231 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:00:27.231 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.231 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:00:27.232 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.232 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "application not enabled on pool 'test'" 2020-08-24T01:00:27.232 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.232 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.232 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-08-24T01:00:27.232 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.233 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.233 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:00:27.233 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.233 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.233 INFO:tasks.workunit.client.0.smithi060.stdout: "mutes": [] 2020-08-24T01:00:27.234 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.234 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_first_committed": 1, 2020-08-24T01:00:27.234 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_last_committed": 1, 2020-08-24T01:00:27.234 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap": { 2020-08-24T01:00:27.235 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:00:27.235 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T01:00:27.235 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:58:00.856785Z", 2020-08-24T01:00:27.235 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:58:00.856785Z", 2020-08-24T01:00:27.236 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release": 15, 2020-08-24T01:00:27.236 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release_name": "octopus", 2020-08-24T01:00:27.237 INFO:tasks.workunit.client.0.smithi060.stdout: "features": { 2020-08-24T01:00:27.237 INFO:tasks.workunit.client.0.smithi060.stdout: "persistent": [ 2020-08-24T01:00:27.237 INFO:tasks.workunit.client.0.smithi060.stdout: "kraken", 2020-08-24T01:00:27.237 INFO:tasks.workunit.client.0.smithi060.stdout: "luminous", 2020-08-24T01:00:27.238 INFO:tasks.workunit.client.0.smithi060.stdout: "mimic", 2020-08-24T01:00:27.238 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap-prune", 2020-08-24T01:00:27.238 INFO:tasks.workunit.client.0.smithi060.stdout: "nautilus", 2020-08-24T01:00:27.238 INFO:tasks.workunit.client.0.smithi060.stdout: "octopus" 2020-08-24T01:00:27.238 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.239 INFO:tasks.workunit.client.0.smithi060.stdout: "optional": [] 2020-08-24T01:00:27.239 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.239 INFO:tasks.workunit.client.0.smithi060.stdout: "mons": [ 2020-08-24T01:00:27.239 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.239 INFO:tasks.workunit.client.0.smithi060.stdout: "rank": 0, 2020-08-24T01:00:27.239 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "a", 2020-08-24T01:00:27.240 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:00:27.240 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.240 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.240 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.240 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115", 2020-08-24T01:00:27.241 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 0 2020-08-24T01:00:27.241 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.241 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.241 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.241 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115/0", 2020-08-24T01:00:27.242 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:7115/0", 2020-08-24T01:00:27.242 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 0, 2020-08-24T01:00:27.242 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 0 2020-08-24T01:00:27.242 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.242 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.243 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.243 INFO:tasks.workunit.client.0.smithi060.stdout: "quorum": [ 2020-08-24T01:00:27.243 INFO:tasks.workunit.client.0.smithi060.stdout: 0 2020-08-24T01:00:27.243 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.243 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap": { 2020-08-24T01:00:27.244 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 43, 2020-08-24T01:00:27.244 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T01:00:27.244 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T00:58:01.596369+0000", 2020-08-24T01:00:27.244 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:59:16.194710+0000", 2020-08-24T01:00:27.244 INFO:tasks.workunit.client.0.smithi060.stdout: "last_up_change": "2020-08-24T00:59:14.574570+0000", 2020-08-24T01:00:27.245 INFO:tasks.workunit.client.0.smithi060.stdout: "last_in_change": "2020-08-24T00:59:15.911184+0000", 2020-08-24T01:00:27.245 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-08-24T01:00:27.245 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_num": 5800384, 2020-08-24T01:00:27.245 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_set": [ 2020-08-24T01:00:27.246 INFO:tasks.workunit.client.0.smithi060.stdout: "nodown", 2020-08-24T01:00:27.246 INFO:tasks.workunit.client.0.smithi060.stdout: "noin", 2020-08-24T01:00:27.246 INFO:tasks.workunit.client.0.smithi060.stdout: "noout", 2020-08-24T01:00:27.247 INFO:tasks.workunit.client.0.smithi060.stdout: "pglog_hardlimit", 2020-08-24T01:00:27.247 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snapdirs", 2020-08-24T01:00:27.247 INFO:tasks.workunit.client.0.smithi060.stdout: "recovery_deletes", 2020-08-24T01:00:27.248 INFO:tasks.workunit.client.0.smithi060.stdout: "sortbitwise" 2020-08-24T01:00:27.248 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.248 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_version": 7, 2020-08-24T01:00:27.248 INFO:tasks.workunit.client.0.smithi060.stdout: "full_ratio": 0.99000000953674316, 2020-08-24T01:00:27.248 INFO:tasks.workunit.client.0.smithi060.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-08-24T01:00:27.249 INFO:tasks.workunit.client.0.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2020-08-24T01:00:27.249 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_snapshot": "", 2020-08-24T01:00:27.249 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_max": 1, 2020-08-24T01:00:27.249 INFO:tasks.workunit.client.0.smithi060.stdout: "max_osd": 3, 2020-08-24T01:00:27.249 INFO:tasks.workunit.client.0.smithi060.stdout: "require_min_compat_client": "jewel", 2020-08-24T01:00:27.250 INFO:tasks.workunit.client.0.smithi060.stdout: "min_compat_client": "jewel", 2020-08-24T01:00:27.250 INFO:tasks.workunit.client.0.smithi060.stdout: "require_osd_release": "octopus", 2020-08-24T01:00:27.250 INFO:tasks.workunit.client.0.smithi060.stdout: "pools": [ 2020-08-24T01:00:27.250 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.250 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T01:00:27.251 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_name": "test", 2020-08-24T01:00:27.251 INFO:tasks.workunit.client.0.smithi060.stdout: "create_time": "2020-08-24T00:58:24.119074+0000", 2020-08-24T01:00:27.251 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": 1, 2020-08-24T01:00:27.251 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_names": "hashpspool", 2020-08-24T01:00:27.251 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T01:00:27.252 INFO:tasks.workunit.client.0.smithi060.stdout: "size": 3, 2020-08-24T01:00:27.252 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 2, 2020-08-24T01:00:27.252 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_rule": 0, 2020-08-24T01:00:27.252 INFO:tasks.workunit.client.0.smithi060.stdout: "object_hash": 2, 2020-08-24T01:00:27.252 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_autoscale_mode": "on", 2020-08-24T01:00:27.253 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num": 32, 2020-08-24T01:00:27.253 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num": 1, 2020-08-24T01:00:27.253 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num_target": 32, 2020-08-24T01:00:27.253 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_target": 32, 2020-08-24T01:00:27.253 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_pending": 32, 2020-08-24T01:00:27.254 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pg_merge_meta": { 2020-08-24T01:00:27.254 INFO:tasks.workunit.client.0.smithi060.stdout: "source_pgid": "0.0", 2020-08-24T01:00:27.254 INFO:tasks.workunit.client.0.smithi060.stdout: "ready_epoch": 0, 2020-08-24T01:00:27.254 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_started": 0, 2020-08-24T01:00:27.255 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": 0, 2020-08-24T01:00:27.255 INFO:tasks.workunit.client.0.smithi060.stdout: "source_version": "0'0", 2020-08-24T01:00:27.255 INFO:tasks.workunit.client.0.smithi060.stdout: "target_version": "0'0" 2020-08-24T01:00:27.255 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.256 INFO:tasks.workunit.client.0.smithi060.stdout: "last_change": "32", 2020-08-24T01:00:27.256 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend": "0", 2020-08-24T01:00:27.256 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2020-08-24T01:00:27.257 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_preluminous": "32", 2020-08-24T01:00:27.257 INFO:tasks.workunit.client.0.smithi060.stdout: "auid": 0, 2020-08-24T01:00:27.258 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_mode": "selfmanaged", 2020-08-24T01:00:27.258 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_seq": 0, 2020-08-24T01:00:27.258 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_epoch": 0, 2020-08-24T01:00:27.258 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_snaps": [], 2020-08-24T01:00:27.259 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps": "[]", 2020-08-24T01:00:27.259 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_bytes": 0, 2020-08-24T01:00:27.259 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_objects": 0, 2020-08-24T01:00:27.259 INFO:tasks.workunit.client.0.smithi060.stdout: "tiers": [], 2020-08-24T01:00:27.259 INFO:tasks.workunit.client.0.smithi060.stdout: "tier_of": -1, 2020-08-24T01:00:27.260 INFO:tasks.workunit.client.0.smithi060.stdout: "read_tier": -1, 2020-08-24T01:00:27.260 INFO:tasks.workunit.client.0.smithi060.stdout: "write_tier": -1, 2020-08-24T01:00:27.260 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_mode": "none", 2020-08-24T01:00:27.260 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_bytes": 0, 2020-08-24T01:00:27.260 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_objects": 0, 2020-08-24T01:00:27.260 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-08-24T01:00:27.261 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-08-24T01:00:27.261 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2020-08-24T01:00:27.261 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_flush_age": 0, 2020-08-24T01:00:27.261 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_evict_age": 0, 2020-08-24T01:00:27.262 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profile": "", 2020-08-24T01:00:27.262 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_params": { 2020-08-24T01:00:27.262 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "none" 2020-08-24T01:00:27.262 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.263 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_period": 0, 2020-08-24T01:00:27.263 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_count": 0, 2020-08-24T01:00:27.263 INFO:tasks.workunit.client.0.smithi060.stdout: "use_gmt_hitset": true, 2020-08-24T01:00:27.263 INFO:tasks.workunit.client.0.smithi060.stdout: "min_read_recency_for_promote": 0, 2020-08-24T01:00:27.264 INFO:tasks.workunit.client.0.smithi060.stdout: "min_write_recency_for_promote": 0, 2020-08-24T01:00:27.264 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2020-08-24T01:00:27.264 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_search_last_n": 0, 2020-08-24T01:00:27.264 INFO:tasks.workunit.client.0.smithi060.stdout: "grade_table": [], 2020-08-24T01:00:27.264 INFO:tasks.workunit.client.0.smithi060.stdout: "stripe_width": 0, 2020-08-24T01:00:27.265 INFO:tasks.workunit.client.0.smithi060.stdout: "expected_num_objects": 0, 2020-08-24T01:00:27.265 INFO:tasks.workunit.client.0.smithi060.stdout: "fast_read": false, 2020-08-24T01:00:27.265 INFO:tasks.workunit.client.0.smithi060.stdout: "options": {}, 2020-08-24T01:00:27.265 INFO:tasks.workunit.client.0.smithi060.stdout: "application_metadata": {} 2020-08-24T01:00:27.265 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.266 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.266 INFO:tasks.workunit.client.0.smithi060.stdout: "osds": [ 2020-08-24T01:00:27.266 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.266 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:00:27.267 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "765fa77d-4249-474d-a0a3-ac21e9faf855", 2020-08-24T01:00:27.267 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T01:00:27.267 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:00:27.267 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:00:27.270 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:00:27.270 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 5, 2020-08-24T01:00:27.270 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 30, 2020-08-24T01:00:27.270 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 38, 2020-08-24T01:00:27.271 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 42, 2020-08-24T01:00:27.271 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 33, 2020-08-24T01:00:27.271 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:00:27.271 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:00:27.271 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.272 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.272 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.272 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6802", 2020-08-24T01:00:27.272 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.272 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.273 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.273 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.273 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6803", 2020-08-24T01:00:27.273 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.274 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.274 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.274 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.274 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:00:27.274 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.275 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.275 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.275 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6804", 2020-08-24T01:00:27.275 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.275 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.276 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.276 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.276 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6805", 2020-08-24T01:00:27.276 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.276 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.276 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.277 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.277 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:00:27.277 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.277 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.277 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.278 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6808", 2020-08-24T01:00:27.278 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.278 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.278 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.278 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.279 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6809", 2020-08-24T01:00:27.279 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.279 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.279 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.279 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.280 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:00:27.280 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.280 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.280 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.280 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6806", 2020-08-24T01:00:27.280 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.281 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.281 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.281 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.281 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6807", 2020-08-24T01:00:27.281 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33546 2020-08-24T01:00:27.282 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.282 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.282 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.282 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6803/33546", 2020-08-24T01:00:27.282 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6805/33546", 2020-08-24T01:00:27.283 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6809/33546", 2020-08-24T01:00:27.283 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6807/33546", 2020-08-24T01:00:27.283 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:00:27.283 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T01:00:27.283 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T01:00:27.283 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.284 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.284 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.284 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:00:27.284 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "b9f5bd36-904e-4fe9-8925-445d22957285", 2020-08-24T01:00:27.284 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 0, 2020-08-24T01:00:27.285 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:00:27.285 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:00:27.285 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:00:27.285 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 0, 2020-08-24T01:00:27.285 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 0, 2020-08-24T01:00:27.285 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 10, 2020-08-24T01:00:27.286 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 22, 2020-08-24T01:00:27.286 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 34, 2020-08-24T01:00:27.286 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:00:27.286 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:00:27.286 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.287 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.287 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.288 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T01:00:27.288 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.288 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.288 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.288 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.288 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T01:00:27.289 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.289 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.289 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.289 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.289 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:00:27.290 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.290 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.290 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.290 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T01:00:27.290 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.291 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.291 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.291 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.291 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T01:00:27.291 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.291 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.292 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.292 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.292 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:00:27.292 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.292 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.293 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.293 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T01:00:27.293 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.293 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.293 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.294 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.294 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T01:00:27.294 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.294 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.294 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.294 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.295 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:00:27.295 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.295 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.295 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.295 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T01:00:27.296 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.296 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.296 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.296 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.296 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T01:00:27.297 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 29117 2020-08-24T01:00:27.297 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.297 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.297 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.297 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/29117", 2020-08-24T01:00:27.297 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/29117", 2020-08-24T01:00:27.298 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/29117", 2020-08-24T01:00:27.298 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/29117", 2020-08-24T01:00:27.298 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:00:27.298 INFO:tasks.workunit.client.0.smithi060.stdout: "exists" 2020-08-24T01:00:27.298 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.299 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.299 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.299 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:00:27.299 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "03903bbb-6dd8-4f53-b407-7ee8ec7022ce", 2020-08-24T01:00:27.299 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T01:00:27.299 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:00:27.300 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:00:27.300 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:00:27.300 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 15, 2020-08-24T01:00:27.300 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 30, 2020-08-24T01:00:27.300 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 39, 2020-08-24T01:00:27.301 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 0, 2020-08-24T01:00:27.301 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 35, 2020-08-24T01:00:27.301 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:00:27.301 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:00:27.301 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.302 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.302 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.302 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T01:00:27.302 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.302 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.302 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.303 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.303 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T01:00:27.303 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.303 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.303 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.304 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.304 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:00:27.304 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.304 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.304 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.305 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T01:00:27.305 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.305 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.305 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.305 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.305 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T01:00:27.306 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.306 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.306 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.307 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.307 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:00:27.307 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.307 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.308 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.308 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T01:00:27.308 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.308 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.308 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.308 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.309 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T01:00:27.309 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.309 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.309 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.309 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.310 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:00:27.310 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:00:27.310 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.310 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:00:27.310 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T01:00:27.311 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.311 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.311 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.311 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:00:27.311 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T01:00:27.311 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 33740 2020-08-24T01:00:27.312 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.312 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.312 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.312 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/33740", 2020-08-24T01:00:27.312 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/33740", 2020-08-24T01:00:27.313 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/33740", 2020-08-24T01:00:27.313 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/33740", 2020-08-24T01:00:27.313 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:00:27.313 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T01:00:27.313 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T01:00:27.313 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.314 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.314 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.314 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_xinfo": [ 2020-08-24T01:00:27.314 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.314 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:00:27.315 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:59:08.073077+0000", 2020-08-24T01:00:27.315 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:00:27.315 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:00:27.315 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:00:27.315 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:00:27.315 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:58:08.576478+0000", 2020-08-24T01:00:27.316 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:00:27.316 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.316 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.316 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:00:27.316 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:59:08.621845+0000", 2020-08-24T01:00:27.317 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:00:27.317 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:00:27.317 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:00:27.317 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:00:27.317 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:58:15.105595+0000", 2020-08-24T01:00:27.318 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:00:27.318 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.318 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.318 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:00:27.318 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T00:59:09.139471+0000", 2020-08-24T01:00:27.318 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:00:27.319 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:00:27.319 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:00:27.319 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:00:27.319 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T00:58:21.928359+0000", 2020-08-24T01:00:27.319 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:00:27.320 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.320 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.320 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap": [], 2020-08-24T01:00:27.320 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap_items": [], 2020-08-24T01:00:27.320 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_temp": [], 2020-08-24T01:00:27.320 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_temp": [], 2020-08-24T01:00:27.321 INFO:tasks.workunit.client.0.smithi060.stdout: "blacklist": {}, 2020-08-24T01:00:27.321 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profiles": { 2020-08-24T01:00:27.321 INFO:tasks.workunit.client.0.smithi060.stdout: "default": { 2020-08-24T01:00:27.321 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T01:00:27.321 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "2", 2020-08-24T01:00:27.322 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T01:00:27.322 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van" 2020-08-24T01:00:27.322 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.322 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.322 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps_queue": [], 2020-08-24T01:00:27.323 INFO:tasks.workunit.client.0.smithi060.stdout: "new_removed_snaps": [], 2020-08-24T01:00:27.323 INFO:tasks.workunit.client.0.smithi060.stdout: "new_purged_snaps": [], 2020-08-24T01:00:27.323 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_node_flags": {}, 2020-08-24T01:00:27.323 INFO:tasks.workunit.client.0.smithi060.stdout: "device_class_flags": {} 2020-08-24T01:00:27.323 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.323 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_metadata": [ 2020-08-24T01:00:27.324 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.324 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:00:27.324 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:00:27.324 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6804/33546,v1:127.0.0.1:6805/33546]", 2020-08-24T01:00:27.324 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:00:27.325 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:00:27.325 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:00:27.325 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:00:27.325 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:00:27.325 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:00:27.325 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:00:27.326 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:00:27.326 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:00:27.326 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-08-24T01:00:27.326 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:00:27.326 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:00:27.327 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:00:27.327 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:00:27.327 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:00:27.327 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:00:27.328 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:00:27.328 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:00:27.328 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:00:27.328 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:00:27.328 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:00:27.329 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:00:27.329 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:00:27.329 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:00:27.329 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:00:27.329 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6802/33546,v1:127.0.0.1:6803/33546]", 2020-08-24T01:00:27.329 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:00:27.330 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/33546,v1:127.0.0.1:6809/33546]", 2020-08-24T01:00:27.330 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/33546,v1:127.0.0.1:6807/33546]", 2020-08-24T01:00:27.330 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:00:27.330 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:00:27.330 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:00:27.331 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:00:27.331 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:00:27.331 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:00:27.331 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:00:27.331 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:00:27.332 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:00:27.332 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/0", 2020-08-24T01:00:27.332 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:00:27.332 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:00:27.332 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:00:27.332 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.333 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.333 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:00:27.333 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:00:27.333 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/29117,v1:127.0.0.1:6813/29117]", 2020-08-24T01:00:27.333 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:00:27.334 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:00:27.334 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:00:27.334 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:00:27.334 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:00:27.334 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:00:27.335 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:00:27.335 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:00:27.335 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:00:27.335 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-08-24T01:00:27.335 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:00:27.335 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:00:27.336 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:00:27.336 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:00:27.336 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:00:27.336 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:00:27.336 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:00:27.337 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:00:27.337 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:00:27.337 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:00:27.337 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:00:27.337 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:00:27.337 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:00:27.338 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:00:27.338 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:00:27.338 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/29117,v1:127.0.0.1:6811/29117]", 2020-08-24T01:00:27.338 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:00:27.338 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/29117,v1:127.0.0.1:6817/29117]", 2020-08-24T01:00:27.339 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/29117,v1:127.0.0.1:6815/29117]", 2020-08-24T01:00:27.339 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:00:27.339 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:00:27.339 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:00:27.339 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:00:27.339 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:00:27.340 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:00:27.340 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:00:27.340 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:00:27.340 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:00:27.340 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/1", 2020-08-24T01:00:27.341 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:00:27.341 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:00:27.341 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:00:27.341 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.341 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.342 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:00:27.342 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:00:27.342 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/33740,v1:127.0.0.1:6813/33740]", 2020-08-24T01:00:27.342 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:00:27.342 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:00:27.342 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:00:27.343 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:00:27.343 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:00:27.343 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:00:27.343 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:00:27.343 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:00:27.344 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:00:27.344 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-08-24T01:00:27.344 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:00:27.344 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:00:27.344 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:00:27.344 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:00:27.345 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:00:27.345 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:00:27.345 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:00:27.345 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:00:27.345 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:00:27.346 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:00:27.346 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:00:27.346 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:00:27.346 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:00:27.346 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:00:27.347 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:00:27.347 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/33740,v1:127.0.0.1:6811/33740]", 2020-08-24T01:00:27.347 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:00:27.347 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/33740,v1:127.0.0.1:6817/33740]", 2020-08-24T01:00:27.348 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/33740,v1:127.0.0.1:6815/33740]", 2020-08-24T01:00:27.348 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:00:27.348 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:00:27.348 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:00:27.349 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:00:27.349 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:00:27.349 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:00:27.349 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:00:27.349 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:00:27.350 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:00:27.350 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/2", 2020-08-24T01:00:27.350 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:00:27.350 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:00:27.350 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:00:27.351 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.351 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.351 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_clean_epochs": { 2020-08-24T01:00:27.351 INFO:tasks.workunit.client.0.smithi060.stdout: "min_last_epoch_clean": 0, 2020-08-24T01:00:27.351 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": { 2020-08-24T01:00:27.352 INFO:tasks.workunit.client.0.smithi060.stdout: "per_pool": [] 2020-08-24T01:00:27.352 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.352 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_epochs": [ 2020-08-24T01:00:27.352 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.352 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:00:27.353 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 38 2020-08-24T01:00:27.353 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.353 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.353 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:00:27.353 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 10 2020-08-24T01:00:27.353 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.354 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.354 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:00:27.354 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 39 2020-08-24T01:00:27.354 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.354 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.355 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.355 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_first_committed": 1, 2020-08-24T01:00:27.356 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_last_committed": 43, 2020-08-24T01:00:27.356 INFO:tasks.workunit.client.0.smithi060.stdout: "crushmap": { 2020-08-24T01:00:27.356 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": [ 2020-08-24T01:00:27.356 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.356 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:00:27.357 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.0", 2020-08-24T01:00:27.357 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:00:27.357 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.357 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.357 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:00:27.358 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.1", 2020-08-24T01:00:27.358 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:00:27.358 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.358 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.358 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:00:27.358 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.2", 2020-08-24T01:00:27.359 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:00:27.359 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.359 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.359 INFO:tasks.workunit.client.0.smithi060.stdout: "types": [ 2020-08-24T01:00:27.359 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.360 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 0, 2020-08-24T01:00:27.360 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd" 2020-08-24T01:00:27.360 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.360 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.360 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:00:27.360 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "host" 2020-08-24T01:00:27.361 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.361 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.361 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 2, 2020-08-24T01:00:27.361 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "chassis" 2020-08-24T01:00:27.361 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.362 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.362 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 3, 2020-08-24T01:00:27.362 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "rack" 2020-08-24T01:00:27.362 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.362 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.363 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 4, 2020-08-24T01:00:27.363 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "row" 2020-08-24T01:00:27.363 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.363 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.363 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 5, 2020-08-24T01:00:27.363 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pdu" 2020-08-24T01:00:27.364 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.364 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.364 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 6, 2020-08-24T01:00:27.364 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pod" 2020-08-24T01:00:27.364 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.365 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.365 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 7, 2020-08-24T01:00:27.365 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "room" 2020-08-24T01:00:27.366 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.366 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.366 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 8, 2020-08-24T01:00:27.366 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "datacenter" 2020-08-24T01:00:27.367 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.367 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.367 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 9, 2020-08-24T01:00:27.367 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "zone" 2020-08-24T01:00:27.367 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.367 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.368 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 10, 2020-08-24T01:00:27.368 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "region" 2020-08-24T01:00:27.368 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.368 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.368 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:00:27.369 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "root" 2020-08-24T01:00:27.369 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.369 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.370 INFO:tasks.workunit.client.0.smithi060.stdout: "buckets": [ 2020-08-24T01:00:27.370 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.370 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -1, 2020-08-24T01:00:27.370 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default", 2020-08-24T01:00:27.370 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:00:27.371 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T01:00:27.371 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:00:27.371 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:00:27.371 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:00:27.372 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:00:27.372 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.372 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T01:00:27.372 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:00:27.372 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:00:27.372 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.373 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.373 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.373 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.373 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -2, 2020-08-24T01:00:27.373 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default~hdd", 2020-08-24T01:00:27.374 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:00:27.374 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T01:00:27.374 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:00:27.374 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:00:27.374 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:00:27.375 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:00:27.375 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.375 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T01:00:27.375 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:00:27.375 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:00:27.375 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.376 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.376 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.376 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.376 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T01:00:27.376 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060", 2020-08-24T01:00:27.377 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:00:27.377 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T01:00:27.377 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:00:27.377 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:00:27.377 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:00:27.377 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:00:27.378 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.378 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:00:27.378 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:00:27.378 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:00:27.378 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.379 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.379 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:00:27.379 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:00:27.380 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T01:00:27.380 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.380 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.380 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:00:27.380 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:00:27.381 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T01:00:27.381 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.381 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.381 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.381 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.381 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T01:00:27.382 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060~hdd", 2020-08-24T01:00:27.382 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:00:27.382 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T01:00:27.382 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:00:27.382 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:00:27.383 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:00:27.383 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:00:27.383 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.383 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:00:27.383 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:00:27.384 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:00:27.384 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.384 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.384 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:00:27.384 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:00:27.384 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T01:00:27.385 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.385 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.385 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:00:27.385 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:00:27.385 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T01:00:27.387 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.387 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.387 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.387 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.387 INFO:tasks.workunit.client.0.smithi060.stdout: "rules": [ 2020-08-24T01:00:27.388 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.388 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 0, 2020-08-24T01:00:27.388 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "replicated_rule", 2020-08-24T01:00:27.388 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 0, 2020-08-24T01:00:27.389 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T01:00:27.389 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 1, 2020-08-24T01:00:27.389 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 10, 2020-08-24T01:00:27.390 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T01:00:27.390 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.390 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T01:00:27.390 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T01:00:27.390 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T01:00:27.391 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.391 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.391 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_firstn", 2020-08-24T01:00:27.391 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T01:00:27.391 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T01:00:27.391 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.392 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.392 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T01:00:27.392 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.392 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:00:27.392 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.393 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.393 INFO:tasks.workunit.client.0.smithi060.stdout: "tunables": { 2020-08-24T01:00:27.393 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_tries": 0, 2020-08-24T01:00:27.393 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_fallback_tries": 0, 2020-08-24T01:00:27.393 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_total_tries": 50, 2020-08-24T01:00:27.394 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_descend_once": 1, 2020-08-24T01:00:27.394 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_vary_r": 1, 2020-08-24T01:00:27.394 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_stable": 1, 2020-08-24T01:00:27.394 INFO:tasks.workunit.client.0.smithi060.stdout: "straw_calc_version": 1, 2020-08-24T01:00:27.394 INFO:tasks.workunit.client.0.smithi060.stdout: "allowed_bucket_algs": 54, 2020-08-24T01:00:27.394 INFO:tasks.workunit.client.0.smithi060.stdout: "profile": "jewel", 2020-08-24T01:00:27.395 INFO:tasks.workunit.client.0.smithi060.stdout: "optimal_tunables": 1, 2020-08-24T01:00:27.395 INFO:tasks.workunit.client.0.smithi060.stdout: "legacy_tunables": 0, 2020-08-24T01:00:27.395 INFO:tasks.workunit.client.0.smithi060.stdout: "minimum_required_version": "jewel", 2020-08-24T01:00:27.395 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables": 1, 2020-08-24T01:00:27.395 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables2": 1, 2020-08-24T01:00:27.396 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v2_rules": 0, 2020-08-24T01:00:27.396 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables3": 1, 2020-08-24T01:00:27.396 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v3_rules": 0, 2020-08-24T01:00:27.396 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v4_buckets": 1, 2020-08-24T01:00:27.396 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables5": 1, 2020-08-24T01:00:27.396 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v5_rules": 0 2020-08-24T01:00:27.397 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.397 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_args": {} 2020-08-24T01:00:27.397 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.397 INFO:tasks.workunit.client.0.smithi060.stdout: "fsmap": { 2020-08-24T01:00:27.397 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:00:27.398 INFO:tasks.workunit.client.0.smithi060.stdout: "default_fscid": -1, 2020-08-24T01:00:27.398 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": { 2020-08-24T01:00:27.398 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": {}, 2020-08-24T01:00:27.398 INFO:tasks.workunit.client.0.smithi060.stdout: "ro_compat": {}, 2020-08-24T01:00:27.398 INFO:tasks.workunit.client.0.smithi060.stdout: "incompat": { 2020-08-24T01:00:27.399 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_1": "base v0.20", 2020-08-24T01:00:27.399 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_2": "client writeable ranges", 2020-08-24T01:00:27.399 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_3": "default file layouts on dirs", 2020-08-24T01:00:27.399 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_4": "dir inode in separate object", 2020-08-24T01:00:27.399 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_5": "mds uses versioned encoding", 2020-08-24T01:00:27.400 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_6": "dirfrag is stored in omap", 2020-08-24T01:00:27.400 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_8": "no anchor table", 2020-08-24T01:00:27.400 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_9": "file layout v2", 2020-08-24T01:00:27.400 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_10": "snaprealm v2" 2020-08-24T01:00:27.401 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.401 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.401 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_flags": { 2020-08-24T01:00:27.401 INFO:tasks.workunit.client.0.smithi060.stdout: "enable_multiple": false, 2020-08-24T01:00:27.401 INFO:tasks.workunit.client.0.smithi060.stdout: "ever_enabled_multiple": false 2020-08-24T01:00:27.402 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.402 INFO:tasks.workunit.client.0.smithi060.stdout: "standbys": [], 2020-08-24T01:00:27.402 INFO:tasks.workunit.client.0.smithi060.stdout: "filesystems": [] 2020-08-24T01:00:27.402 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.402 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_first_committed": 1, 2020-08-24T01:00:27.402 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_last_committed": 1, 2020-08-24T01:00:27.403 INFO:tasks.workunit.client.0.smithi060.stdout: "auth": { 2020-08-24T01:00:27.403 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T01:00:27.403 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 5, 2020-08-24T01:00:27.403 INFO:tasks.workunit.client.0.smithi060.stdout: "num_secrets": 3 2020-08-24T01:00:27.403 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.404 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 32, 2020-08-24T01:00:27.404 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_active": 32, 2020-08-24T01:00:27.404 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_unknown": 0, 2020-08-24T01:00:27.404 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osd": 3, 2020-08-24T01:00:27.404 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_sum": { 2020-08-24T01:00:27.404 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T01:00:27.405 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61854, 2020-08-24T01:00:27.405 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T01:00:27.405 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T01:00:27.405 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T01:00:27.405 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T01:00:27.406 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T01:00:27.406 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T01:00:27.406 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T01:00:27.406 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T01:00:27.406 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T01:00:27.407 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T01:00:27.407 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T01:00:27.407 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T01:00:27.407 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T01:00:27.407 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T01:00:27.407 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T01:00:27.408 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T01:00:27.408 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T01:00:27.408 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T01:00:27.408 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T01:00:27.408 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T01:00:27.409 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T01:00:27.409 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T01:00:27.409 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T01:00:27.409 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T01:00:27.409 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T01:00:27.409 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T01:00:27.410 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T01:00:27.410 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T01:00:27.410 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T01:00:27.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T01:00:27.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T01:00:27.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T01:00:27.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T01:00:27.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T01:00:27.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T01:00:27.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T01:00:27.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T01:00:27.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T01:00:27.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T01:00:27.412 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.413 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T01:00:27.413 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T01:00:27.413 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T01:00:27.413 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T01:00:27.413 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 0, 2020-08-24T01:00:27.414 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 0, 2020-08-24T01:00:27.414 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:00:27.414 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:00:27.414 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:00:27.414 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T01:00:27.414 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T01:00:27.415 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.415 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 513, 2020-08-24T01:00:27.415 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 513, 2020-08-24T01:00:27.415 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 64, 2020-08-24T01:00:27.415 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 64, 2020-08-24T01:00:27.416 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 0 2020-08-24T01:00:27.416 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.416 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum": { 2020-08-24T01:00:27.416 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T01:00:27.416 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T01:00:27.416 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 65, 2020-08-24T01:00:27.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T01:00:27.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T01:00:27.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T01:00:27.417 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T01:00:27.417 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3180288, 2020-08-24T01:00:27.418 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 34240, 2020-08-24T01:00:27.418 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 6, 2020-08-24T01:00:27.418 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145721, 2020-08-24T01:00:27.418 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311392512, 2020-08-24T01:00:27.418 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T01:00:27.419 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T01:00:27.419 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318865932288, 2020-08-24T01:00:27.419 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T01:00:27.419 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 35061760, 2020-08-24T01:00:27.419 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 480504, 2020-08-24T01:00:27.419 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:00:27.420 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:00:27.420 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:00:27.420 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 6646, 2020-08-24T01:00:27.420 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221218826 2020-08-24T01:00:27.421 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.421 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T01:00:27.421 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T01:00:27.421 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T01:00:27.421 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T01:00:27.428 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T01:00:27.428 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T01:00:27.429 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T01:00:27.429 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.429 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T01:00:27.429 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T01:00:27.429 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T01:00:27.430 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T01:00:27.430 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T01:00:27.430 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.431 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T01:00:27.432 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T01:00:27.432 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.432 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum_by_class": { 2020-08-24T01:00:27.432 INFO:tasks.workunit.client.0.smithi060.stdout: "hdd": { 2020-08-24T01:00:27.432 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T01:00:27.432 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T01:00:27.433 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 65, 2020-08-24T01:00:27.433 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T01:00:27.433 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T01:00:27.433 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T01:00:27.433 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T01:00:27.434 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3180288, 2020-08-24T01:00:27.434 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 34240, 2020-08-24T01:00:27.434 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 6, 2020-08-24T01:00:27.434 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145721, 2020-08-24T01:00:27.435 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311392512, 2020-08-24T01:00:27.435 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T01:00:27.435 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T01:00:27.435 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318865932288, 2020-08-24T01:00:27.435 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T01:00:27.436 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 35061760, 2020-08-24T01:00:27.436 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 480504, 2020-08-24T01:00:27.436 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:00:27.436 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:00:27.436 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:00:27.437 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 6646, 2020-08-24T01:00:27.437 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221218826 2020-08-24T01:00:27.437 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.437 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T01:00:27.437 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T01:00:27.437 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T01:00:27.438 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T01:00:27.438 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T01:00:27.438 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T01:00:27.438 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T01:00:27.439 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.439 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T01:00:27.439 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T01:00:27.439 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T01:00:27.439 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T01:00:27.440 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T01:00:27.440 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.440 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T01:00:27.440 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T01:00:27.440 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.441 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.441 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_stats": [ 2020-08-24T01:00:27.441 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.441 INFO:tasks.workunit.client.0.smithi060.stdout: "poolid": 1, 2020-08-24T01:00:27.442 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 32, 2020-08-24T01:00:27.442 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T01:00:27.442 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 61854, 2020-08-24T01:00:27.442 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 100, 2020-08-24T01:00:27.442 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 0, 2020-08-24T01:00:27.443 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 300, 2020-08-24T01:00:27.443 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T01:00:27.443 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T01:00:27.443 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 100, 2020-08-24T01:00:27.443 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T01:00:27.443 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T01:00:27.444 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 100, 2020-08-24T01:00:27.444 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T01:00:27.444 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T01:00:27.444 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T01:00:27.444 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T01:00:27.445 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T01:00:27.445 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T01:00:27.445 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T01:00:27.445 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T01:00:27.445 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T01:00:27.446 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T01:00:27.446 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T01:00:27.446 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T01:00:27.446 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T01:00:27.446 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T01:00:27.446 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T01:00:27.447 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T01:00:27.447 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T01:00:27.447 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T01:00:27.447 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T01:00:27.447 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T01:00:27.448 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T01:00:27.448 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T01:00:27.448 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T01:00:27.448 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T01:00:27.448 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T01:00:27.448 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T01:00:27.449 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T01:00:27.449 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T01:00:27.449 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T01:00:27.449 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T01:00:27.449 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.450 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T01:00:27.450 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T01:00:27.450 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T01:00:27.450 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T01:00:27.450 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 19529728, 2020-08-24T01:00:27.451 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 184482, 2020-08-24T01:00:27.451 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:00:27.451 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:00:27.451 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:00:27.451 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T01:00:27.451 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T01:00:27.452 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.452 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 513, 2020-08-24T01:00:27.452 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 513, 2020-08-24T01:00:27.452 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 64, 2020-08-24T01:00:27.452 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 64, 2020-08-24T01:00:27.453 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 3 2020-08-24T01:00:27.453 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.453 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.453 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_stats": [ 2020-08-24T01:00:27.453 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.454 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:00:27.454 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 163208757267 2020-08-24T01:00:27.454 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.454 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.454 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:00:27.454 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 42949672977 2020-08-24T01:00:27.455 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.455 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.455 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:00:27.455 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 167503724563 2020-08-24T01:00:27.455 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.456 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.456 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_state": [ 2020-08-24T01:00:27.456 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.456 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "active+undersized+degraded", 2020-08-24T01:00:27.456 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 32 2020-08-24T01:00:27.456 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.457 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.457 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_osd": [ 2020-08-24T01:00:27.457 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.457 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:00:27.457 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:00:27.458 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 32, 2020-08-24T01:00:27.458 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:00:27.458 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.458 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.458 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:00:27.459 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:00:27.459 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 0, 2020-08-24T01:00:27.459 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:00:27.459 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.459 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.459 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:00:27.460 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 32, 2020-08-24T01:00:27.460 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 32, 2020-08-24T01:00:27.460 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:00:27.460 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.460 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.461 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": [ 2020-08-24T01:00:27.461 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:00:27.461 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T01:00:27.461 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": {} 2020-08-24T01:00:27.461 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.462 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:00:27.462 INFO:tasks.workunit.client.0.smithi060.stdout: "servicemap": { 2020-08-24T01:00:27.462 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:00:27.462 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T00:58:01.597183+0000", 2020-08-24T01:00:27.462 INFO:tasks.workunit.client.0.smithi060.stdout: "services": {} 2020-08-24T01:00:27.462 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:00:27.463 INFO:tasks.workunit.client.0.smithi060.stdout: "paxos": { 2020-08-24T01:00:27.463 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T01:00:27.463 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 451, 2020-08-24T01:00:27.463 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pn": 0, 2020-08-24T01:00:27.463 INFO:tasks.workunit.client.0.smithi060.stdout: "accepted_pn": 0 2020-08-24T01:00:27.464 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:00:27.464 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:00:27.465 INFO:tasks.workunit.client.0.smithi060.stderr:report 119733822 2020-08-24T01:00:27.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-08-24T01:00:27.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-08-24T01:00:27.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:00:27.466 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery 2020-08-24T01:00:27.541 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:00:27.541 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596654+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.541 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595585+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.542 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596617+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.542 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.598040+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.542 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595917+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.542 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498491+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.543 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498693+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.543 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595671+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.543 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597434+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.543 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595309+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.543 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594225+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.544 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593998+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.544 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595105+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.544 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596904+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.544 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594689+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.544 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594715+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.545 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594497+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.545 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594556+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.545 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595667+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.545 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594190+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.545 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596910+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.546 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595302+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.546 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596159+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.546 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498563+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.546 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595022+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.547 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593526+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.547 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597416+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.547 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593709+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.547 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593683+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.547 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596397+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.547 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 0 3 0 0 2082 0 0 17 17 active+undersized+degraded 2020-08-24T00:59:16.594050+0000 43'101 43:125 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.548 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595020+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:27.548 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:00:27.548 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:00:27.549 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:00:27.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-08-24T01:00:27.552 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-08-24T01:00:27.553 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T01:00:27.553 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T01:00:27.554 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:00:27.554 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:00:27.554 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:00:27.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T01:00:27.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-08-24T01:00:27.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok set_recovery_delay 100000 2020-08-24T01:00:27.652 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T01:00:27.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-08-24T01:00:27.660 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-08-24T01:00:27.660 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T01:00:27.660 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T01:00:27.661 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:00:27.661 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:00:27.662 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:00:27.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T01:00:27.662 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-08-24T01:00:27.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok set_recovery_delay 100000 2020-08-24T01:00:27.762 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T01:00:27.771 INFO:tasks.workunit.client.0.smithi060.stdout:revive divergent 1 2020-08-24T01:00:27.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-08-24T01:00:27.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:00:28.126 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:00:28.126 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596654+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.127 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595585+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.127 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596617+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.127 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.598040+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.127 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595917+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.128 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498491+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.128 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498693+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.128 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595671+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.128 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597434+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.128 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595309+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.129 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594225+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.129 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593998+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.129 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595105+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.134 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596904+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.134 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594689+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.134 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594715+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.134 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594497+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.135 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594556+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.135 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595667+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.135 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594190+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.135 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596910+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.135 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595302+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.136 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596159+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.136 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498563+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.136 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595022+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.136 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593526+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.136 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597416+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.137 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593709+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.137 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593683+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.137 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596397+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.137 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 0 3 0 0 2082 0 0 17 17 active+undersized+degraded 2020-08-24T00:59:16.594050+0000 43'101 43:125 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.138 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595020+0000 30'100 43:124 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:28.138 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:00:28.138 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:00:28.139 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:00:28.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-08-24T01:00:28.822 INFO:tasks.workunit.client.0.smithi060.stderr:noup is set 2020-08-24T01:00:28.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-08-24T01:00:28.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:00:28.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:00:28.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T01:00:28.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:00:28.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T01:00:28.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:00:28.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:00:28.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:00:28.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:00:28.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T01:00:28.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T01:00:28.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:00:28.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:00:28.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:00:28.838 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:00:28.839 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:00:28.839 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:00:28.842 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:00:28.843 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:00:28.843 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:00:28.843 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:00:28.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:00:28.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:00:28.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:00:28.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:00:28.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:00:28.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:00:28.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:00:28.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:00:28.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:00:28.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T01:00:28.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T01:00:28.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:00:28.847 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T01:00:28.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T01:00:28.852 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:00:28.852 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:00:28.852 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:00:28.852 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:00:28.870 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:28.866+0000 7f872c8a9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:00:28.872 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:28.870+0000 7f872c8a9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:00:28.874 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:28.870+0000 7f872c8a9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:00:29.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-08-24T01:00:36.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-08-24T01:00:36.129 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:29.406+0000 7f872c8a9d80 -1 Falling back to public interface 2020-08-24T01:00:36.129 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:30.378+0000 7f872c8a9d80 -1 osd.1 31 log_to_monitors {default=true} 2020-08-24T01:00:36.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-08-24T01:00:36.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:00:36.130 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:00:36.130 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-08-24T01:00:36.130 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:00:36.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:00:36.131 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:00:36.131 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:00:36.131 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:00:36.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T01:00:36.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-08-24T01:00:36.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.1.asok set_recovery_delay 100000 2020-08-24T01:00:36.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-08-24T01:00:36.132 INFO:tasks.workunit.client.0.smithi060.stderr:noup is unset 2020-08-24T01:00:36.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-08-24T01:00:36.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:00:36.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:00:36.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:00:36.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:00:36.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:00:36.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:00:36.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:00:36.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:00:36.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:00:36.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:00:36.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:00:36.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-08-24T01:00:36.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:00:36.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:00:36.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:00:36.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:00:36.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:00:36.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:00:36.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:00:36.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:00:36.137 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery divergent 2020-08-24T01:00:36.137 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:00:36.137 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596654+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.137 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595585+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.137 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596617+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.140 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.598040+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.140 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595917+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.140 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498491+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.140 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498693+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.141 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595671+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.141 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597434+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.141 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595309+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.141 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594225+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.141 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593998+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.142 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595105+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.142 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596904+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.142 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594689+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.142 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594715+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.142 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594497+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.143 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594556+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.143 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595667+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.143 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.594190+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.143 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596910+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.143 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 4 0 0 1929 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595302+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.144 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596159+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.144 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.498563+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.144 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595022+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.144 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593526+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.144 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.597416+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.145 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593709+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.145 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.593683+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.145 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.596397+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.145 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 0 3 0 0 2082 0 0 17 17 active+undersized+degraded 2020-08-24T00:59:16.594050+0000 43'101 44:127 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.145 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 3 0 0 1928 0 0 16 16 active+undersized+degraded 2020-08-24T00:59:16.595020+0000 30'100 44:126 [0,2] 0 [0,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:36.146 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:00:36.146 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:00:36.146 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T01:00:36.146 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:00:36.146 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 38 up_thru 42 down_at 33 last_clean_interval [5,30) [v2:127.0.0.1:6802/33546,v1:127.0.0.1:6803/33546] [v2:127.0.0.1:6804/33546,v1:127.0.0.1:6805/33546] exists,up 765fa77d-4249-474d-a0a3-ac21e9faf855 2020-08-24T01:00:36.147 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:00:36.296 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 34 last_clean_interval [10,31) [v2:127.0.0.1:6818/36214,v1:127.0.0.1:6819/36214] [v2:127.0.0.1:6820/36214,v1:127.0.0.1:6821/36214] exists,up b9f5bd36-904e-4fe9-8925-445d22957285 2020-08-24T01:00:36.296 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:00:36.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:00:36.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:00:36.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:00:36.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-08-24T01:00:36.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:00:36.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:00:36.298 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:00:36.298 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:00:36.298 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:00:36.298 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:00:36.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:00:36.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:00:36.671 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 39 up_thru 0 down_at 35 last_clean_interval [15,30) [v2:127.0.0.1:6810/33740,v1:127.0.0.1:6811/33740] [v2:127.0.0.1:6812/33740,v1:127.0.0.1:6813/33740] exists,up 03903bbb-6dd8-4f53-b407-7ee8ec7022ce 2020-08-24T01:00:36.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:00:36.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:00:36.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:00:36.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:00:37.025 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:00:37.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.277114+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.281827+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278250+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.027 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278211+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.027 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278154+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.027 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279978+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.027 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.288511+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.027 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.280031+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.028 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.280008+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.028 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.280028+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.028 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279102+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.028 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.288016+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.029 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.282224+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.029 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.282079+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.029 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.282254+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.029 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278633+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.031 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.286071+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.032 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.284412+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.032 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.286316+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.032 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.284434+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.032 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279746+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.033 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.278202+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.033 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.277783+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.033 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.287299+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.033 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.288129+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.034 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.287292+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.034 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.277894+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.034 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279554+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.034 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.286532+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.034 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.284192+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.035 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.286539+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.035 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.283978+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.035 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:00:37.035 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:00:37.036 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:00:37.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-08-24T01:00:37.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:00:37.037 INFO:tasks.workunit.client.0.smithi060.stdout:wait for peering 2020-08-24T01:00:37.390 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:00:37.390 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.277114+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.391 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.281827+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.391 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278250+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.391 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278211+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.391 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278154+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.391 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279978+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.392 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.288511+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.392 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.280031+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.392 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.280008+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.392 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.280028+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.392 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279102+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.393 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.288016+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.393 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.282224+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.393 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.282079+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.393 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.282254+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.394 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.278633+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.394 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.286071+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.394 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.284412+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.394 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.286316+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.394 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.284434+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.395 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279746+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.395 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.278202+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.395 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 0 0 0 1900 0 0 16 16 peering 2020-08-24T01:00:35.277783+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.395 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.287299+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.395 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.288129+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.396 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.287292+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.396 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.277894+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.396 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.279554+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.396 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.286532+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.396 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.284192+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.397 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.286539+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.397 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 0 0 0 1899 0 0 16 16 peering 2020-08-24T01:00:35.283978+0000 30'106 47:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:00:37.397 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:00:37.397 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:00:37.398 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:00:37.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-08-24T01:00:44.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-08-24T01:01:09.957 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:01:09.960 INFO:tasks.workunit.client.0.smithi060.stdout:killing divergent 1 2020-08-24T01:01:09.980 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:01:09.981 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.902048+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.981 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.903488+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.981 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.906225+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.981 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.905654+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.981 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 1 1 0 0 1928 0 0 10 10 active+recovery_wait+degraded+wait 2020-08-24T01:00:35.901231+0000 30'100 48:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.984 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.873190+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.984 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.898202+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.984 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.905812+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.985 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.905188+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.985 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.900864+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.985 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.900795+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.985 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.900044+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.986 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.905554+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.996 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.904473+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.996 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 1 1 0 0 1928 0 0 16 16 active+recovering+degraded+wait 2020-08-24T01:00:35.907069+0000 30'100 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.997 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.901074+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.997 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.902805+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.997 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.905289+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.997 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.903416+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.998 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.900040+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.999 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.903965+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.999 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 0 0 0 1929 0 0 15 15 active+clean+wait 2020-08-24T01:00:35.901365+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:09.999 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.901644+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.000 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.839208+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.001 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.897897+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.001 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 1 1 0 0 1928 0 0 11 11 active+recovery_wait+degraded+wait 2020-08-24T01:00:35.901759+0000 30'100 48:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.002 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.904351+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.002 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.897627+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.002 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean+wait 2020-08-24T01:00:35.897506+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.003 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 0 0 0 1928 0 0 12 12 active+clean+wait 2020-08-24T01:00:35.903903+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.003 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 1 1 0 0 2082 0 0 14 14 active+recovery_wait+degraded+wait 2020-08-24T01:00:35.904999+0000 43'101 48:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.003 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.903115+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.004 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:01:10.004 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:01:10.004 INFO:tasks.workunit.client.0.smithi060.stdout: marking collection for removal 2020-08-24T01:01:10.004 INFO:tasks.workunit.client.0.smithi060.stdout:setting '_remove' omap key 2020-08-24T01:01:10.005 INFO:tasks.workunit.client.0.smithi060.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-08-24T01:01:10.005 INFO:tasks.workunit.client.0.smithi060.stdout:get_pg_num_history pg_num_history pg_num_history(e48 pg_nums {1={22=1,32=32}} deleted_pools ) 2020-08-24T01:01:10.005 INFO:tasks.workunit.client.0.smithi060.stdout:Importing pgid 1.0 2020-08-24T01:01:10.006 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:023dd3d2:::existing_12:head# 2020-08-24T01:01:10.006 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T01:01:10.006 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:04c40a0c:::existing_93:head# 2020-08-24T01:01:10.007 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T01:01:10.008 INFO:tasks.workunit.client.0.smithi060.stdout:Write #1:064ff5fb:::existing_36:head# 2020-08-24T01:01:10.009 INFO:tasks.workunit.client.0.smithi060.stdout:snapset 0=[]:{} 2020-08-24T01:01:10.009 INFO:tasks.workunit.client.0.smithi060.stdout:write_pg epoch 48 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/48 n=3 ec=22/22 lis/c=47/47 les/c/f=48/48/0 sis=47 pruub=8.696315490s) 2020-08-24T01:01:10.009 INFO:tasks.workunit.client.0.smithi060.stdout:Import successful 2020-08-24T01:01:10.009 INFO:tasks.workunit.client.0.smithi060.stdout:reviving divergent 1 2020-08-24T01:01:10.009 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:01:10.010 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.902048+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.010 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.903488+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.010 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.906225+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.010 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.905654+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.010 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 1 1 0 0 1928 0 0 10 10 active+recovery_wait+degraded+wait 2020-08-24T01:00:35.901231+0000 30'100 48:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.011 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.873190+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.011 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.898202+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.011 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.905812+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.012 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.905188+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.012 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.900864+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.012 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.900795+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.012 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.900044+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.013 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.905554+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.013 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.904473+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.013 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 1 1 0 0 1928 0 0 16 16 active+recovering+degraded+wait 2020-08-24T01:00:35.907069+0000 30'100 48:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.014 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.901074+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.014 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.902805+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.014 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.905289+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.014 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.903416+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.014 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 0 0 0 1929 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.900040+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.015 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.903965+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.015 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 0 0 0 1929 0 0 15 15 active+clean+wait 2020-08-24T01:00:35.901365+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.015 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.901644+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.015 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.839208+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.015 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.897897+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.016 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 3 1 1 0 0 1928 0 0 11 11 active+recovery_wait+degraded+wait 2020-08-24T01:00:35.901759+0000 30'100 48:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.016 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.904351+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.016 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean+wait 2020-08-24T01:00:35.897627+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.016 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean+wait 2020-08-24T01:00:35.897506+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.016 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 0 0 0 1928 0 0 12 12 active+clean+wait 2020-08-24T01:00:35.903903+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.017 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 1 1 0 0 2082 0 0 14 14 active+recovery_wait+degraded+wait 2020-08-24T01:00:35.904999+0000 43'101 48:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.017 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean+wait 2020-08-24T01:00:35.903115+0000 30'100 48:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.017 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:01:10.017 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:01:10.018 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:01:10.018 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:01:10.018 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 34 last_clean_interval [10,31) [v2:127.0.0.1:6818/36214,v1:127.0.0.1:6819/36214] [v2:127.0.0.1:6820/36214,v1:127.0.0.1:6821/36214] exists,up b9f5bd36-904e-4fe9-8925-445d22957285 2020-08-24T01:01:10.019 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:01:10.019 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 47 up_thru 47 down_at 34 last_clean_interval [10,31) [v2:127.0.0.1:6818/36214,v1:127.0.0.1:6819/36214] [v2:127.0.0.1:6820/36214,v1:127.0.0.1:6821/36214] exists,up b9f5bd36-904e-4fe9-8925-445d22957285 2020-08-24T01:01:10.019 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:01:10.019 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [], 2020-08-24T01:01:10.019 INFO:tasks.workunit.client.0.smithi060.stdout: "num_ops": 0 2020-08-24T01:01:10.020 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:01:10.020 INFO:tasks.workunit.client.0.smithi060.stdout:allowing recovery 2020-08-24T01:01:10.020 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:01:10.020 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.691604+0000 30'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.020 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-08-24T01:01:01.352768+0000 30'100 57:146 [0,1,2] 0 [0,1,2] 0 30'100 2020-08-24T01:01:01.352638+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.021 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.531168+0000 30'100 58:128 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.021 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:50.751824+0000 30'100 57:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.021 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:01:00.686955+0000 30'100 57:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.021 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:58.659018+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 30'100 2020-08-24T01:00:52.593201+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.021 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.529004+0000 30'100 58:128 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:00:54.505561+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.022 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:00:50.751014+0000 30'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.022 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.688534+0000 30'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.022 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:58.658637+0000 30'100 57:143 [0,1,2] 0 [0,1,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.022 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-08-24T01:00:57.679072+0000 30'100 58:131 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:00:57.678949+0000 30'100 2020-08-24T01:00:57.678949+0000 0 2020-08-24T01:01:10.022 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:00:50.750295+0000 30'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.023 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:01:00.686222+0000 30'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.023 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:58.657062+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.023 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.527069+0000 30'100 58:128 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.023 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:00:50.752658+0000 30'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.023 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.528205+0000 30'100 58:128 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.024 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 2 0 0 0 0 1234 0 0 16 16 active+clean 2020-08-24T01:01:00.528234+0000 30'100 57:147 [1,0,2] 1 [1,0,2] 1 30'100 2020-08-24T01:01:00.528119+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.024 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean 2020-08-24T01:01:00.688151+0000 30'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.024 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 0 0 0 1929 0 0 16 16 active+clean 2020-08-24T01:00:58.658418+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 30'100 2020-08-24T01:00:57.412944+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.024 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:00:50.752712+0000 30'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.024 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 4 0 0 0 0 1929 0 0 16 16 active+clean 2020-08-24T01:00:56.526514+0000 30'100 58:128 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.025 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 4 0 0 0 0 2545 0 0 17 17 active+clean 2020-08-24T01:00:58.659974+0000 48'101 57:144 [0,1,2] 0 [0,1,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.025 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.686181+0000 30'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.025 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-08-24T01:01:01.448168+0000 30'100 57:148 [1,0,2] 1 [1,0,2] 1 30'100 2020-08-24T01:01:01.448042+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.025 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 5 0 0 0 0 3085 0 0 16 16 active+clean 2020-08-24T01:01:02.591116+0000 30'100 58:131 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:01:02.590982+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:58.657018+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:01:00.690407+0000 30'100 57:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean 2020-08-24T01:00:50.752849+0000 30'100 57:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.530184+0000 30'100 58:128 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 3 0 0 0 0 2082 0 0 17 17 active+clean 2020-08-24T01:00:58.581558+0000 43'101 57:144 [0,1,2] 0 [0,1,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.026 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.611185+0000 30'100 57:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.027 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:01:10.027 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:01:10.027 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:01:10.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-08-24T01:01:10.028 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:01:10.028 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:01:10.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:01:10.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:01:10.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:01:10.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:01:10.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.11819.out 2020-08-24T01:01:10.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.11819.out 2020-08-24T01:01:10.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-08-24T01:01:10.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:01:10.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-08-24T01:01:10.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:01:10.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-08-24T01:01:10.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.11819.out 2020-08-24T01:01:10.031 INFO:tasks.workunit.client.0.smithi060.stderr:Exporting 1.0 info 1.0( v 30'100 (30'90,30'100] local-lis/les=47/48 n=3 ec=22/22 lis/c=47/47 les/c/f=48/48/0 sis=47 pruub=8.696315490s) 2020-08-24T01:01:10.032 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:023dd3d2:::existing_12:head# 2020-08-24T01:01:10.032 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:04c40a0c:::existing_93:head# 2020-08-24T01:01:10.032 INFO:tasks.workunit.client.0.smithi060.stderr:Read #1:064ff5fb:::existing_36:head# 2020-08-24T01:01:10.032 INFO:tasks.workunit.client.0.smithi060.stderr:Export successful 2020-08-24T01:01:10.032 INFO:tasks.workunit.client.0.smithi060.stderr:Remove successful 2020-08-24T01:01:10.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.11819.out 2020-08-24T01:01:10.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-08-24T01:01:10.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:01:10.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-08-24T01:01:10.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:01:10.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-08-24T01:01:10.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.11819.out 2020-08-24T01:01:10.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-08-24T01:01:10.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:01:10.035 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:01:10.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-08-24T01:01:10.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:01:10.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:01:10.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T01:01:10.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:01:10.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T01:01:10.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:01:10.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:01:10.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:01:10.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:01:10.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T01:01:10.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T01:01:10.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:01:10.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:01:10.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:01:10.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:01:10.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:01:10.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:01:10.039 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:01:10.039 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:10.039 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:10.039 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:10.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:10.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:01:10.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:01:10.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:01:10.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:01:10.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:01:10.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:01:10.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:01:10.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:01:10.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T01:01:10.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T01:01:10.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:01:10.050 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T01:01:10.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T01:01:10.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:01:10.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:01:10.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:01:10.051 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:47.982+0000 7f705cc46d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:10.051 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:47.986+0000 7f705cc46d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:10.052 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:47.986+0000 7f705cc46d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:10.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-08-24T01:01:10.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:01:10.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:01:10.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:01:10.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:01:10.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:01:10.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:01:10.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:01:10.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:01:10.053 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:48.522+0000 7f705cc46d80 -1 Falling back to public interface 2020-08-24T01:01:10.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:01:10.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:01:10.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:01:10.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-08-24T01:01:10.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:01:10.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:01:10.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:01:10.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:01:10.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:01:10.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:01:10.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:01:10.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:01:10.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:01:10.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:01:10.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:01:10.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-08-24T01:01:10.059 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:00:49.582+0000 7f705cc46d80 -1 osd.1 48 log_to_monitors {default=true} 2020-08-24T01:01:10.059 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-08-24T01:01:10.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:01:10.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:01:10.060 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:01:10.060 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:10.060 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:10.061 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T01:01:10.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-08-24T01:01:10.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.1.asok dump_ops_in_flight 2020-08-24T01:01:10.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-08-24T01:01:10.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:01:10.062 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:01:10.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-08-24T01:01:10.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-08-24T01:01:10.062 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T01:01:10.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-08-24T01:01:10.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-08-24T01:01:10.063 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T01:01:10.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-08-24T01:01:10.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-08-24T01:01:10.215 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T01:01:10.226 INFO:tasks.workunit.client.0.smithi060.stdout:reading divergent objects 2020-08-24T01:01:10.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-08-24T01:01:10.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-08-24T01:01:10.628 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:01:10.629 INFO:tasks.workunit.client.0.smithi060.stdout:1.1f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.691604+0000 30'100 61:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.629 INFO:tasks.workunit.client.0.smithi060.stdout:1.1e 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-08-24T01:01:06.615057+0000 30'100 61:153 [1,0,2] 1 [1,0,2] 1 30'100 2020-08-24T01:01:01.352638+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.629 INFO:tasks.workunit.client.0.smithi060.stdout:1.1d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.531168+0000 30'100 61:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.629 INFO:tasks.workunit.client.0.smithi060.stdout:1.1c 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:50.751824+0000 30'100 61:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.630 INFO:tasks.workunit.client.0.smithi060.stdout:1.1b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:01:00.686955+0000 30'100 61:157 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.630 INFO:tasks.workunit.client.0.smithi060.stdout:1.1a 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:58.659018+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 30'100 2020-08-24T01:00:52.593201+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.630 INFO:tasks.workunit.client.0.smithi060.stdout:1.19 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.529004+0000 30'100 61:131 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:00:54.505561+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.630 INFO:tasks.workunit.client.0.smithi060.stdout:1.18 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:00:50.751014+0000 30'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.630 INFO:tasks.workunit.client.0.smithi060.stdout:1.17 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.688534+0000 30'100 61:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.631 INFO:tasks.workunit.client.0.smithi060.stdout:1.16 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:06.581240+0000 30'100 61:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.631 INFO:tasks.workunit.client.0.smithi060.stdout:1.15 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-08-24T01:00:57.679072+0000 30'100 61:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:00:57.678949+0000 30'100 2020-08-24T01:00:57.678949+0000 0 2020-08-24T01:01:10.631 INFO:tasks.workunit.client.0.smithi060.stdout:1.14 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-08-24T01:01:07.502407+0000 30'100 61:152 [1,0,2] 1 [1,0,2] 1 30'100 2020-08-24T01:01:07.502352+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.631 INFO:tasks.workunit.client.0.smithi060.stdout:1.13 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:01:00.686222+0000 30'100 61:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.631 INFO:tasks.workunit.client.0.smithi060.stdout:1.12 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:58.657062+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.632 INFO:tasks.workunit.client.0.smithi060.stdout:1.11 3 0 0 0 0 1851 0 0 16 16 active+clean 2020-08-24T01:01:05.537866+0000 30'100 61:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:01:05.537734+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.632 INFO:tasks.workunit.client.0.smithi060.stdout:1.10 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:00:50.752658+0000 30'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.632 INFO:tasks.workunit.client.0.smithi060.stdout:1.5 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.528205+0000 30'100 61:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.632 INFO:tasks.workunit.client.0.smithi060.stdout:1.4 2 0 0 0 0 1234 0 0 16 16 active+clean 2020-08-24T01:01:00.528234+0000 30'100 61:151 [1,0,2] 1 [1,0,2] 1 30'100 2020-08-24T01:01:00.528119+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.632 INFO:tasks.workunit.client.0.smithi060.stdout:1.3 4 0 0 0 0 1929 0 0 10 10 active+clean 2020-08-24T01:01:00.688151+0000 30'100 61:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.633 INFO:tasks.workunit.client.0.smithi060.stdout:1.2 4 0 0 0 0 1929 0 0 16 16 active+clean 2020-08-24T01:00:58.658418+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 30'100 2020-08-24T01:00:57.412944+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.633 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:00:50.752712+0000 30'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.633 INFO:tasks.workunit.client.0.smithi060.stdout:1.1 2 0 0 0 0 1234 0 0 16 16 active+clean 2020-08-24T01:01:06.582020+0000 30'100 61:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:01:06.581899+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.633 INFO:tasks.workunit.client.0.smithi060.stdout:1.6 4 0 0 0 0 2545 0 0 17 17 active+clean 2020-08-24T01:01:06.613612+0000 48'101 61:153 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.634 INFO:tasks.workunit.client.0.smithi060.stdout:1.7 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.686181+0000 30'100 61:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.635 INFO:tasks.workunit.client.0.smithi060.stdout:1.8 3 0 0 0 0 1851 0 0 10 10 active+clean 2020-08-24T01:01:01.448168+0000 30'100 61:152 [1,0,2] 1 [1,0,2] 1 30'100 2020-08-24T01:01:01.448042+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.635 INFO:tasks.workunit.client.0.smithi060.stdout:1.9 5 0 0 0 0 3085 0 0 16 16 active+clean 2020-08-24T01:01:02.591116+0000 30'100 61:134 [2,0,1] 2 [2,0,1] 2 30'100 2020-08-24T01:01:02.590982+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.635 INFO:tasks.workunit.client.0.smithi060.stdout:1.a 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:58.657018+0000 30'100 60:146 [0,1,2] 0 [0,1,2] 0 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.635 INFO:tasks.workunit.client.0.smithi060.stdout:1.b 3 0 0 0 0 1928 0 0 10 10 active+clean 2020-08-24T01:01:00.690407+0000 30'100 61:155 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.635 INFO:tasks.workunit.client.0.smithi060.stdout:1.c 3 0 0 0 0 1928 0 0 14 14 active+clean 2020-08-24T01:00:50.752849+0000 30'100 61:149 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.636 INFO:tasks.workunit.client.0.smithi060.stdout:1.d 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:00:56.530184+0000 30'100 61:131 [2,0,1] 2 [2,0,1] 2 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.636 INFO:tasks.workunit.client.0.smithi060.stdout:1.e 4 0 0 0 0 2622 0 0 14 14 active+clean 2020-08-24T01:01:06.614704+0000 43'101 61:155 [1,0,2] 1 [1,0,2] 1 43'101 2020-08-24T01:01:03.470540+0000 43'101 2020-08-24T01:01:03.470540+0000 0 2020-08-24T01:01:10.636 INFO:tasks.workunit.client.0.smithi060.stdout:1.f 3 0 0 0 0 1928 0 0 16 16 active+clean 2020-08-24T01:01:00.611185+0000 30'100 61:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-08-24T00:58:24.169278+0000 0'0 2020-08-24T00:58:24.169278+0000 0 2020-08-24T01:01:10.636 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:01:10.636 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:01:10.637 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:01:10.646 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-08-24T01:01:10.647 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-08-24T01:01:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:01:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-08-24T01:01:10.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:01:10.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-08-24T01:01:10.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:01:10.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-08-24T01:01:10.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:01:10.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-08-24T01:01:10.838 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-08-24T01:01:10.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T01:01:10.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-08-24T01:01:10.878 INFO:tasks.workunit.client.0.smithi060.stderr:error getting test/newobject_1: (2) No such file or directory 2020-08-24T01:01:10.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-08-24T01:01:10.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-08-24T01:01:10.925 INFO:tasks.workunit.client.0.smithi060.stderr:error getting test/newobject_2: (2) No such file or directory 2020-08-24T01:01:10.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-08-24T01:01:10.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-08-24T01:01:10.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T01:01:10.942 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-08-24T00:58:57.254158+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.942 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 20 _merge_object_divergent_entries: keeping 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-08-24T00:58:57.254158+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.942 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:01:10.943 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 30'2 first_divergent_update: 30'104 last_divergent_update: 30'104 2020-08-24T01:01:10.943 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-08-24T01:01:10.943 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-08-24T01:01:10.943 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 30'106 2020-08-24T01:01:10.943 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 30'104 (30'2) modify 1:97db2a0f:::existing_2:head by client.4540.0:1 2020-08-24T00:58:57.254158+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.944 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-08-24T01:01:10.944 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.710+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 30'2 <= info.log_tail 30'89 2020-08-24T01:01:10.944 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 30'105 (30'4) delete 1:70b3bf12:::existing_4:head by client.4542.0:1 2020-08-24T00:58:57.256143+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T01:01:10.944 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 20 _merge_object_divergent_entries: keeping 30'105 (30'4) delete 1:70b3bf12:::existing_4:head by client.4542.0:1 2020-08-24T00:58:57.256143+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T01:01:10.945 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:01:10.945 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 30'4 first_divergent_update: 30'105 last_divergent_update: 30'105 2020-08-24T01:01:10.945 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 10 _merge_object_divergent_entries: more recent entry found: 43'101 (30'4) modify 1:70b3bf12:::existing_4:head by client.4652.0:1 2020-08-24T00:59:51.850389+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0, already merged 2020-08-24T01:01:10.945 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 30'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4529.0:1 2020-08-24T00:58:57.241133+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.945 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 20 _merge_object_divergent_entries: keeping 30'101 (0'0) modify 1:75ac5091:::newobject_2:head by client.4529.0:1 2020-08-24T00:58:57.241133+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.950 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:01:10.950 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 30'101 last_divergent_update: 30'101 2020-08-24T01:01:10.950 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-08-24T01:01:10.951 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.714+0000 7f87094e2700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-08-24T01:01:10.951 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-08-24T00:58:57.257006+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T01:01:10.951 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 20 _merge_object_divergent_entries: keeping 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-08-24T00:58:57.257006+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T01:01:10.951 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-08-24T01:01:10.952 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 30'3 first_divergent_update: 30'106 last_divergent_update: 30'106 2020-08-24T01:01:10.952 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-08-24T01:01:10.952 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-08-24T01:01:10.952 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 30'106 2020-08-24T01:01:10.953 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 30'106 (30'3) delete 1:8881e6d8:::existing_3:head by client.4543.0:1 2020-08-24T00:58:57.257006+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 0, new_object: 0 2020-08-24T01:01:10.966 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-08-24T01:01:10.967 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 30'3 <= info.log_tail 30'84 2020-08-24T01:01:10.968 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4533.0:1 2020-08-24T00:58:57.249420+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.969 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 20 _merge_object_divergent_entries: keeping 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4533.0:1 2020-08-24T00:58:57.249420+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.969 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:01:10.970 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 30'1 first_divergent_update: 30'102 last_divergent_update: 30'102 2020-08-24T01:01:10.971 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-08-24T01:01:10.971 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-08-24T01:01:10.971 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 30'106 2020-08-24T01:01:10.971 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 30'102 (30'1) modify 1:d858c035:::existing_1:head by client.4533.0:1 2020-08-24T00:58:57.249420+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.972 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-08-24T01:01:10.972 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 30'1 <= info.log_tail 30'90 2020-08-24T01:01:10.972 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 30'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4537.0:1 2020-08-24T00:58:57.251696+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.972 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 20 _merge_object_divergent_entries: keeping 30'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4537.0:1 2020-08-24T00:58:57.251696+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T01:01:10.973 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:01:10.973 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 30'103 last_divergent_update: 30'103 2020-08-24T01:01:10.973 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-08-24T01:01:10.973 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:00:35.718+0000 7f870a4e4700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-08-24T01:01:10.985 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-08-24T01:01:10.986 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T01:01:10.988 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-08-24T01:01:10.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T01:01:10.999 INFO:tasks.workunit.client.0.smithi060.stdout:success 2020-08-24T01:01:10.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-08-24T01:01:10.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.11819.out 2020-08-24T01:01:11.000 INFO:tasks.workunit.client.0.smithi060.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.11819.out': No such file or directory 2020-08-24T01:01:11.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-08-24T01:01:11.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-08-24T01:01:11.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-08-24T01:01:11.700 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' does not exist 2020-08-24T01:01:11.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-08-24T01:01:11.714 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:01:11.714 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:01:11.714 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:01:11.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:01:11.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:01:12.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:01:12.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-08-24T01:01:12.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T01:01:12.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:01:12.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T01:01:12.036 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:01:12.036 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:01:12.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:01:12.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:01:12.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:01:12.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:01:12.046 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:01:12.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:01:12.047 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:01:12.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:01:12.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:01:12.055 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:01:12.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:01:12.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:01:12.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:01:12.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:01:12.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:01:12.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:01:12.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T01:01:12.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:01:12.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:12.079 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:12.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T01:01:12.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:01:12.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:01:12.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-08-24T01:01:12.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-08-24T01:01:12.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-08-24T01:01:12.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-08-24T01:01:12.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T01:01:12.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:01:12.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T01:01:12.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:01:12.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:01:12.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:01:12.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:01:12.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:01:12.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:01:12.086 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:01:12.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:01:12.088 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:01:12.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:01:12.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:01:12.092 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:01:12.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:01:12.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:01:12.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:01:12.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:01:12.095 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:01:12.096 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:01:12.096 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T01:01:12.099 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:01:12.099 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:12.100 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:12.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T01:01:12.101 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:01:12.101 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:01:12.101 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-08-24T01:01:12.103 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:01:12.104 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:12.104 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:12.104 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.11819 2020-08-24T01:01:12.106 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:01:12.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-08-24T01:01:12.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:01:12.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-08-24T01:01:12.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-08-24T01:01:12.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-08-24T01:01:12.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-08-24T01:01:12.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-08-24T01:01:12.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-08-24T01:01:12.118 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-08-24T01:01:12.118 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-08-24T01:01:12.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-08-24T01:01:12.118 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:01:12.118 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T01:01:12.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-08-24T01:01:12.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-08-24T01:01:12.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:01:12.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:01:12.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:01:12.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-08-24T01:01:12.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-08-24T01:01:12.406 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:01:12.406 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:01:12.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:01:12.407 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:01:12.408 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:12.408 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:12.408 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:12.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:01:12.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:01:12.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:01:12.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:01:12.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:01:12.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:01:12.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:01:12.596 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:01:12.598 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:01:12.598 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:01:12.598 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:01:12.598 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:12.598 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:12.599 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T01:01:12.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:01:12.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get fsid 2020-08-24T01:01:12.695 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:01:12.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:01:12.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:01:12.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:01:12.697 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:01:12.698 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:01:12.698 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:01:12.699 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:01:12.699 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:01:12.699 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:12.700 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:12.700 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-mon.a.asok 2020-08-24T01:01:12.700 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:01:12.700 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.11819/ceph-mon.a.asok config get mon_host 2020-08-24T01:01:12.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-08-24T01:01:12.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-08-24T01:01:12.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:01:12.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:01:12.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:01:12.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-08-24T01:01:12.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:01:13.699 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:01:13.700 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:01:13.700 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:01:13.701 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:01:13.701 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:13.702 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:13.702 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:13.702 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:01:13.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:01:13.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-08-24T01:01:13.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-08-24T01:01:13.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T01:01:13.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:01:13.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:01:13.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:01:13.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-08-24T01:01:13.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:01:13.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:01:13.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:01:13.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:01:13.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T01:01:13.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T01:01:13.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:01:13.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:01:13.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:01:13.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:01:13.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:01:13.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:01:13.739 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:01:13.740 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:13.740 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:13.740 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:13.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:13.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:01:13.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:01:13.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:01:13.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:01:13.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:01:13.742 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:01:13.742 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:01:13.742 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:01:13.742 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:01:13.742 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:01:13.743 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-08-24T01:01:13.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:01:13.743 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2590757d-b008-4deb-b066-fccba9726114 2020-08-24T01:01:13.743 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 2590757d-b008-4deb-b066-fccba9726114' 2020-08-24T01:01:13.744 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 2590757d-b008-4deb-b066-fccba9726114 2020-08-24T01:01:13.744 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:01:13.759 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBZEUNfcpw5LRAAw0tvrMFU0tCDtTbq+b52Bw== 2020-08-24T01:01:13.760 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBZEUNfcpw5LRAAw0tvrMFU0tCDtTbq+b52Bw=="}' 2020-08-24T01:01:13.760 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2590757d-b008-4deb-b066-fccba9726114 -i td/divergent-priors/0/new.json 2020-08-24T01:01:14.088 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:01:14.098 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-08-24T01:01:14.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBZEUNfcpw5LRAAw0tvrMFU0tCDtTbq+b52Bw== --osd-uuid 2590757d-b008-4deb-b066-fccba9726114 2020-08-24T01:01:14.139 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.134+0000 7fe08f45bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:14.143 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.138+0000 7fe08f45bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:14.147 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.142+0000 7fe08f45bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:14.147 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.142+0000 7fe08f45bd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T01:01:14.147 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.142+0000 7fe08f45bd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T01:01:14.147 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.142+0000 7fe08f45bd80 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-08-24T01:01:14.194 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.190+0000 7fe08f45bd80 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-08-24T01:01:14.529 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:14.526+0000 7fe08f45bd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:01:17.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-08-24T01:01:17.261 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:01:17.261 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:01:17.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:01:17.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:01:17.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:01:17.764 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:01:17.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:01:17.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:01:17.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:01:17.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:01:17.799 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:17.794+0000 7f02cd976d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:17.821 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:17.818+0000 7f02cd976d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:17.822 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:17.818+0000 7f02cd976d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:18.125 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:01:18.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:01:18.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:01:18.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:01:18.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:01:18.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:01:18.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:01:18.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:01:18.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:01:18.128 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:01:18.358 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:18.354+0000 7f02cd976d80 -1 Falling back to public interface 2020-08-24T01:01:18.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:01:19.124 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:19.118+0000 7f02cd976d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:01:19.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:01:19.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:01:19.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:01:19.507 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:01:19.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:01:19.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:01:19.880 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/37695,v1:127.0.0.1:6803/37695] [v2:127.0.0.1:6804/37695,v1:127.0.0.1:6805/37695] exists,up 2590757d-b008-4deb-b066-fccba9726114 2020-08-24T01:01:19.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:01:19.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:01:19.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:01:19.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-08-24T01:01:19.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-08-24T01:01:19.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T01:01:19.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:01:19.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:01:19.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:01:19.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-08-24T01:01:19.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:01:19.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:01:19.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:01:19.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:01:19.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T01:01:19.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T01:01:19.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:01:19.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:01:19.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:01:19.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:01:19.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:01:19.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:01:19.892 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:01:19.893 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:19.893 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:19.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:19.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:19.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:01:19.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:01:19.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:01:19.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:01:19.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:01:19.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:01:19.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:01:19.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:01:19.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:01:19.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:01:19.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-08-24T01:01:19.896 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:01:19.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=eb457a2e-e0bb-468b-864e-4386438ddbc4 2020-08-24T01:01:19.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 eb457a2e-e0bb-468b-864e-4386438ddbc4' 2020-08-24T01:01:19.896 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:01:19.896 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 eb457a2e-e0bb-468b-864e-4386438ddbc4 2020-08-24T01:01:19.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBfEUNfblBpNhAAw2wj3yPLajrBGlQN4QrN7g== 2020-08-24T01:01:19.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBfEUNfblBpNhAAw2wj3yPLajrBGlQN4QrN7g=="}' 2020-08-24T01:01:19.915 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new eb457a2e-e0bb-468b-864e-4386438ddbc4 -i td/divergent-priors/1/new.json 2020-08-24T01:01:20.393 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:01:20.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-08-24T01:01:20.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBfEUNfblBpNhAAw2wj3yPLajrBGlQN4QrN7g== --osd-uuid eb457a2e-e0bb-468b-864e-4386438ddbc4 2020-08-24T01:01:20.446 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:20.442+0000 7fc6c0c61d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:20.448 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:20.446+0000 7fc6c0c61d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:20.449 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:20.446+0000 7fc6c0c61d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:20.450 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:20.446+0000 7fc6c0c61d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T01:01:20.450 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:20.446+0000 7fc6c0c61d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T01:01:20.451 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:20.446+0000 7fc6c0c61d80 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-08-24T01:01:20.526 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:20.522+0000 7fc6c0c61d80 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-08-24T01:01:21.069 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:21.066+0000 7fc6c0c61d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:01:23.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-08-24T01:01:23.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:01:23.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:01:23.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:01:23.884 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:01:24.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:01:24.357 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:01:24.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:01:24.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:01:24.359 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:01:24.359 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:01:24.394 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:24.390+0000 7f12e6009d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:24.397 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:24.394+0000 7f12e6009d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:24.399 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:24.394+0000 7f12e6009d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:24.712 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:01:24.713 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:01:24.713 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:01:24.713 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:01:24.714 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:01:24.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:01:24.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:01:24.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:01:24.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:01:24.718 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:01:24.942 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:24.934+0000 7f12e6009d80 -1 Falling back to public interface 2020-08-24T01:01:25.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:01:25.717 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:25.714+0000 7f12e6009d80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:01:26.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:01:26.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:01:26.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:01:26.084 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:01:26.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:01:26.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:01:26.466 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/38050,v1:127.0.0.1:6811/38050] [v2:127.0.0.1:6812/38050,v1:127.0.0.1:6813/38050] exists,up eb457a2e-e0bb-468b-864e-4386438ddbc4 2020-08-24T01:01:26.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:01:26.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:01:26.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:01:26.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-08-24T01:01:26.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-08-24T01:01:26.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-08-24T01:01:26.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:01:26.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T01:01:26.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:01:26.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-08-24T01:01:26.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:01:26.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:01:26.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:01:26.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:01:26.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T01:01:26.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T01:01:26.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:01:26.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:01:26.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:01:26.471 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:01:26.471 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:01:26.471 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:01:26.474 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:01:26.474 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:01:26.474 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:01:26.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:26.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:01:26.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:01:26.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:01:26.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:01:26.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:01:26.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:01:26.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:01:26.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:01:26.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:01:26.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:01:26.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:01:26.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-08-24T01:01:26.478 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:01:26.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=813ed60f-f06b-4b40-a01a-9681d9e7dd5d 2020-08-24T01:01:26.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 813ed60f-f06b-4b40-a01a-9681d9e7dd5d' 2020-08-24T01:01:26.478 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:01:26.479 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 813ed60f-f06b-4b40-a01a-9681d9e7dd5d 2020-08-24T01:01:26.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBmEUNf7f1kHRAARfJeZlh8q4Mccrf3djV8JQ== 2020-08-24T01:01:26.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBmEUNf7f1kHRAARfJeZlh8q4Mccrf3djV8JQ=="}' 2020-08-24T01:01:26.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 813ed60f-f06b-4b40-a01a-9681d9e7dd5d -i td/divergent-priors/2/new.json 2020-08-24T01:01:27.030 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:01:27.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-08-24T01:01:27.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBmEUNf7f1kHRAARfJeZlh8q4Mccrf3djV8JQ== --osd-uuid 813ed60f-f06b-4b40-a01a-9681d9e7dd5d 2020-08-24T01:01:27.080 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.078+0000 7f7da3051d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:27.082 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.078+0000 7f7da3051d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:27.084 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.082+0000 7f7da3051d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:01:27.085 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.082+0000 7f7da3051d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T01:01:27.085 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.082+0000 7f7da3051d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T01:01:27.085 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.082+0000 7f7da3051d80 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-08-24T01:01:27.125 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.122+0000 7f7da3051d80 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-08-24T01:01:27.600 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:27.590+0000 7f7da3051d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:01:30.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-08-24T01:03:00.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:03:00.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T01:03:00.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:03:00.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T01:03:00.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:03:00.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:00.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:03:00.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:03:00.469 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:30.994+0000 7f30fc19dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.470 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:30.998+0000 7f30fc19dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.470 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:30.998+0000 7f30fc19dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T01:03:00.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:03:00.470 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:03:00.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:03:00.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:03:00.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:00.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:03:00.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:03:00.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:00.472 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:31.542+0000 7f30fc19dd80 -1 Falling back to public interface 2020-08-24T01:03:00.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:03:00.472 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:01:32.366+0000 7f30fc19dd80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T01:03:00.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:03:00.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:00.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:03:00.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:00.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:03:00.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:03:00.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:03:00.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:03:00.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-08-24T01:03:00.474 INFO:tasks.workunit.client.0.smithi060.stderr:noout is set 2020-08-24T01:03:00.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-08-24T01:03:00.475 INFO:tasks.workunit.client.0.smithi060.stderr:noin is set 2020-08-24T01:03:00.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-08-24T01:03:00.475 INFO:tasks.workunit.client.0.smithi060.stderr:nodown is set 2020-08-24T01:03:00.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-08-24T01:03:00.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: create_ec_pool: local pool_name=test 2020-08-24T01:03:00.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: shift 2020-08-24T01:03:00.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2273: create_ec_pool: local allow_overwrites=true 2020-08-24T01:03:00.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: shift 2020-08-24T01:03:00.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-08-24T01:03:00.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2278: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-08-24T01:03:00.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-08-24T01:03:00.477 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T01:03:00.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:03:00.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: '[' true = true ']' 2020-08-24T01:03:00.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-08-24T01:03:00.478 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 allow_ec_overwrites to true 2020-08-24T01:03:00.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: create_ec_pool: wait_for_clean 2020-08-24T01:03:00.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:03:00.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:03:00.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:03:00.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:03:00.479 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:03:00.479 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:03:00.480 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:03:00.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:03:00.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:03:00.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:03:00.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:03:00.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:03:00.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:03:00.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:03:00.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:03:00.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:03:00.482 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:03:00.482 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T01:03:00.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:03:00.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.482 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:03:00.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-08-24T01:03:00.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836487 2020-08-24T01:03:00.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836487' 2020-08-24T01:03:00.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.483 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:03:00.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-08-24T01:03:00.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705669 2020-08-24T01:03:00.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2020-08-24T01:03:00.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.484 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:03:00.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-08-24T01:03:00.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542148 2020-08-24T01:03:00.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-60129542148' 2020-08-24T01:03:00.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836487 2020-08-24T01:03:00.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:03:00.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836487 2020-08-24T01:03:00.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836487 2020-08-24T01:03:00.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-08-24T01:03:00.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836487 -lt 21474836487 2020-08-24T01:03:00.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705669 2020-08-24T01:03:00.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:03:00.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705669 2020-08-24T01:03:00.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705669 2020-08-24T01:03:00.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-08-24T01:03:00.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:03:00.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705668 -lt 38654705669 2020-08-24T01:03:00.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:03:00.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:03:00.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:03:00.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705669 -lt 38654705669 2020-08-24T01:03:00.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542148 2020-08-24T01:03:00.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:03:00.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.492 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542148 2020-08-24T01:03:00.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542148 2020-08-24T01:03:00.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-08-24T01:03:00.492 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:03:00.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542148 -lt 60129542148 2020-08-24T01:03:00.492 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:03:00.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:03:00.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T01:03:00.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:03:00.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:03:00.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: create_ec_pool: return 0 2020-08-24T01:03:00.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-08-24T01:03:00.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:03:00.497 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:03:00.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:03:00.497 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:03:00.497 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T01:03:00.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:03:00.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:03:00.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-08-24T01:03:00.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836490 2020-08-24T01:03:00.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836490' 2020-08-24T01:03:00.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:03:00.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705672 2020-08-24T01:03:00.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705672 2020-08-24T01:03:00.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836490 1-38654705672' 2020-08-24T01:03:00.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:03:00.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542151 2020-08-24T01:03:00.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542151 2020-08-24T01:03:00.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-60129542151' 2020-08-24T01:03:00.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.501 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836490 2020-08-24T01:03:00.501 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:03:00.501 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-08-24T01:03:00.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836490 2020-08-24T01:03:00.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-08-24T01:03:00.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836488 -lt 21474836490 2020-08-24T01:03:00.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:03:00.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:03:00.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836490 -lt 21474836490 2020-08-24T01:03:00.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705672 2020-08-24T01:03:00.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:03:00.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705672 2020-08-24T01:03:00.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705672 2020-08-24T01:03:00.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2020-08-24T01:03:00.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:03:00.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705672 -lt 38654705672 2020-08-24T01:03:00.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542151 2020-08-24T01:03:00.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:03:00.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542151 2020-08-24T01:03:00.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542151 2020-08-24T01:03:00.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542151' 2020-08-24T01:03:00.507 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:03:00.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542151 -lt 60129542151 2020-08-24T01:03:00.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-08-24T01:03:00.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:03:00.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:03:00.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:03:00.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:03:00.508 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:03:00.508 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:03:00.509 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:03:00.509 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:03:00.509 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:03:00.509 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:03:00.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:03:00.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:03:00.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:03:00.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:03:00.510 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:03:00.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:03:00.511 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:03:00.511 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T01:03:00.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:03:00.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.511 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:03:00.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-08-24T01:03:00.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836493 2020-08-24T01:03:00.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836493' 2020-08-24T01:03:00.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:03:00.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705675 2020-08-24T01:03:00.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705675 2020-08-24T01:03:00.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836493 1-38654705675' 2020-08-24T01:03:00.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.513 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:03:00.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542153 2020-08-24T01:03:00.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542153 2020-08-24T01:03:00.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836493 1-38654705675 2-60129542153' 2020-08-24T01:03:00.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.514 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836493 2020-08-24T01:03:00.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:03:00.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836493 2020-08-24T01:03:00.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836493 2020-08-24T01:03:00.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-08-24T01:03:00.516 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836493 -lt 21474836493 2020-08-24T01:03:00.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.516 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705675 2020-08-24T01:03:00.516 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:03:00.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705675 2020-08-24T01:03:00.517 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705675 2020-08-24T01:03:00.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2020-08-24T01:03:00.518 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:03:00.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705673 -lt 38654705675 2020-08-24T01:03:00.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:03:00.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:03:00.518 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:03:00.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705675 -lt 38654705675 2020-08-24T01:03:00.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542153 2020-08-24T01:03:00.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:03:00.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542153 2020-08-24T01:03:00.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542153 2020-08-24T01:03:00.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2020-08-24T01:03:00.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:03:00.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542154 -lt 60129542153 2020-08-24T01:03:00.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:03:00.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:03:00.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.523 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T01:03:00.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:03:00.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:03:00.524 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-08-24T01:03:00.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-08-24T01:03:00.525 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:03:00.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-08-24T01:03:00.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-08-24T01:03:00.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:03:00.527 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:03:00.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-08-24T01:03:00.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-08-24T01:03:00.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-08-24T01:03:00.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-08-24T01:03:00.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-08-24T01:03:00.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-08-24T01:03:00.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-08-24T01:03:00.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-08-24T01:03:00.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-08-24T01:03:00.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-08-24T01:03:00.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-08-24T01:03:00.530 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-08-24T01:03:00.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-08-24T01:03:00.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-08-24T01:03:00.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-08-24T01:03:00.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-08-24T01:03:00.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-08-24T01:03:00.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-08-24T01:03:00.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-08-24T01:03:00.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-08-24T01:03:00.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-08-24T01:03:00.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-08-24T01:03:00.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-08-24T01:03:00.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-08-24T01:03:00.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-08-24T01:03:00.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-08-24T01:03:00.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-08-24T01:03:00.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-08-24T01:03:00.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-08-24T01:03:00.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-08-24T01:03:00.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-08-24T01:03:00.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-08-24T01:03:00.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-08-24T01:03:00.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-08-24T01:03:00.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-08-24T01:03:00.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-08-24T01:03:00.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-08-24T01:03:00.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-08-24T01:03:00.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-08-24T01:03:00.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-08-24T01:03:00.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-08-24T01:03:00.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-08-24T01:03:00.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-08-24T01:03:00.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-08-24T01:03:00.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-08-24T01:03:00.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-08-24T01:03:00.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-08-24T01:03:00.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-08-24T01:03:00.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-08-24T01:03:00.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-08-24T01:03:00.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-08-24T01:03:00.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-08-24T01:03:00.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-08-24T01:03:00.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-08-24T01:03:00.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-08-24T01:03:00.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-08-24T01:03:00.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-08-24T01:03:00.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-08-24T01:03:00.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-08-24T01:03:00.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-08-24T01:03:00.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-08-24T01:03:00.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-08-24T01:03:00.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-08-24T01:03:00.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-08-24T01:03:00.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-08-24T01:03:00.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-08-24T01:03:00.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-08-24T01:03:00.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-08-24T01:03:00.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-08-24T01:03:00.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-08-24T01:03:00.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-08-24T01:03:00.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-08-24T01:03:00.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-08-24T01:03:00.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-08-24T01:03:00.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-08-24T01:03:00.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-08-24T01:03:00.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-08-24T01:03:00.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-08-24T01:03:00.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-08-24T01:03:00.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-08-24T01:03:00.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-08-24T01:03:00.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-08-24T01:03:00.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-08-24T01:03:00.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-08-24T01:03:00.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-08-24T01:03:00.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-08-24T01:03:00.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-08-24T01:03:00.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-08-24T01:03:00.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-08-24T01:03:00.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-08-24T01:03:00.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-08-24T01:03:00.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-08-24T01:03:00.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-08-24T01:03:00.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-08-24T01:03:00.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-08-24T01:03:00.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-08-24T01:03:00.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-08-24T01:03:00.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-08-24T01:03:00.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-08-24T01:03:00.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-08-24T01:03:00.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-08-24T01:03:00.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-08-24T01:03:00.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-08-24T01:03:00.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-08-24T01:03:00.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-08-24T01:03:00.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-08-24T01:03:00.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-08-24T01:03:00.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-08-24T01:03:00.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-08-24T01:03:00.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-08-24T01:03:00.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-08-24T01:03:00.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-08-24T01:03:00.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-08-24T01:03:00.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T01:03:00.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-08-24T01:03:00.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:03:00.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:03:00.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:03:00.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:03:00.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T01:03:00.581 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:03:00.581 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:03:00.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:03:00.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:03:00.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:03:00.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:03:00.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:03:00.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:03:00.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:03:00.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:03:00.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:03:00.583 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:03:00.583 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T01:03:00.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:03:00.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:03:00.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836500 2020-08-24T01:03:00.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836500 2020-08-24T01:03:00.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836500' 2020-08-24T01:03:00.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:03:00.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705682 2020-08-24T01:03:00.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705682 2020-08-24T01:03:00.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836500 1-38654705682' 2020-08-24T01:03:00.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.586 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:03:00.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542160 2020-08-24T01:03:00.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542160 2020-08-24T01:03:00.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836500 1-38654705682 2-60129542160' 2020-08-24T01:03:00.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836500 2020-08-24T01:03:00.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:03:00.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836500 2020-08-24T01:03:00.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836500 2020-08-24T01:03:00.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836500' 2020-08-24T01:03:00.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836498 -lt 21474836500 2020-08-24T01:03:00.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:03:00.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:03:00.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836500 -lt 21474836500 2020-08-24T01:03:00.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705682 2020-08-24T01:03:00.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:03:00.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705682 2020-08-24T01:03:00.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705682 2020-08-24T01:03:00.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705682' 2020-08-24T01:03:00.591 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:03:00.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705682 -lt 38654705682 2020-08-24T01:03:00.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.592 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542160 2020-08-24T01:03:00.593 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:03:00.593 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.593 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542160 2020-08-24T01:03:00.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542160 2020-08-24T01:03:00.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2020-08-24T01:03:00.594 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:03:00.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542160 -lt 60129542160 2020-08-24T01:03:00.594 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:03:00.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:03:00.597 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.597 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.597 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T01:03:00.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:03:00.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:03:00.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-08-24T01:03:00.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-08-24T01:03:00.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-08-24T01:03:00.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-08-24T01:03:00.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-08-24T01:03:00.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-08-24T01:03:00.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-08-24T01:03:00.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:03:00.600 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:03:00.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-08-24T01:03:00.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T01:03:00.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T01:03:00.601 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:03:00.602 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:03:00.602 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:03:00.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T01:03:00.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-08-24T01:03:00.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok config set objectstore_blackhole 1 2020-08-24T01:03:00.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.603 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-08-24T01:03:00.603 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T01:03:00.603 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T01:03:00.603 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:03:00.603 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:03:00.604 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:03:00.604 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T01:03:00.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-08-24T01:03:00.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok config set objectstore_blackhole 1 2020-08-24T01:03:00.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-08-24T01:03:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-08-24T01:03:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-08-24T01:03:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-08-24T01:03:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-08-24T01:03:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-08-24T01:03:00.606 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-08-24T01:03:00.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-08-24T01:03:00.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-08-24T01:03:00.606 INFO:tasks.workunit.client.0.smithi060.stderr:rados: no process found 2020-08-24T01:03:00.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-08-24T01:03:00.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:03:00.607 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:03:00.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-08-24T01:03:00.607 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:03:00.607 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:03:00.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:03:00.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:03:00.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:03:00.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:03:00.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-08-24T01:03:00.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-08-24T01:03:00.609 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already down. 2020-08-24T01:03:00.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-08-24T01:03:00.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-08-24T01:03:00.610 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already down. 2020-08-24T01:03:00.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-08-24T01:03:00.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-08-24T01:03:00.610 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already down. 2020-08-24T01:03:00.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-08-24T01:03:00.611 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already out. 2020-08-24T01:03:00.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-08-24T01:03:00.612 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already out. 2020-08-24T01:03:00.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-08-24T01:03:00.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:03:00.612 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:03:00.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-08-24T01:03:00.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:03:00.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:03:00.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-08-24T01:03:00.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:03:00.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-08-24T01:03:00.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:03:00.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:03:00.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:03:00.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:03:00.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T01:03:00.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T01:03:00.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:03:00.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:03:00.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:03:00.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:03:00.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:03:00.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:03:00.617 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:03:00.617 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:03:00.618 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:03:00.618 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:00.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:00.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:03:00.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:03:00.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:03:00.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:03:00.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:03:00.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:00.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:03:00.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:03:00.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-08-24T01:03:00.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-08-24T01:03:00.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:00.620 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-08-24T01:03:00.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-08-24T01:03:00.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:03:00.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:03:00.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:03:00.621 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:31.717+0000 7fd8bf8f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.621 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:31.721+0000 7fd8bf8f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.622 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:31.725+0000 7fd8bf8f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-08-24T01:03:00.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:03:00.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:03:00.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:03:00.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:03:00.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:00.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:03:00.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:00.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:03:00.624 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:32.265+0000 7fd8bf8f3d80 -1 Falling back to public interface 2020-08-24T01:03:00.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:03:00.624 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:33.185+0000 7fd8bf8f3d80 -1 osd.0 27 log_to_monitors {default=true} 2020-08-24T01:03:00.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:03:00.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:00.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:03:00.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:00.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:03:00.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:03:00.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:03:00.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:03:00.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-08-24T01:03:00.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:03:00.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:03:00.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-08-24T01:03:00.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:03:00.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-08-24T01:03:00.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:03:00.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:03:00.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:03:00.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:03:00.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T01:03:00.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T01:03:00.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:03:00.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:03:00.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:03:00.630 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:03:00.630 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:03:00.630 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:03:00.630 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:03:00.630 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:03:00.631 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:03:00.631 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:00.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:00.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:03:00.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:03:00.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:03:00.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:03:00.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:03:00.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:00.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:03:00.632 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:03:00.633 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-08-24T01:03:00.633 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-08-24T01:03:00.633 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:00.633 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-08-24T01:03:00.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-08-24T01:03:00.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:03:00.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:03:00.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:03:00.634 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:33.869+0000 7ff7b64b9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.634 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:33.873+0000 7ff7b64b9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.635 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:33.873+0000 7ff7b64b9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:00.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-08-24T01:03:00.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:03:00.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:03:00.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:03:00.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:03:00.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:00.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:03:00.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:00.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:03:00.637 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:34.413+0000 7ff7b64b9d80 -1 Falling back to public interface 2020-08-24T01:03:00.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:03:00.637 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:02:35.389+0000 7ff7b64b9d80 -1 osd.2 27 log_to_monitors {default=true} 2020-08-24T01:03:00.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:03:00.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:00.638 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:03:00.638 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:00.638 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:03:00.638 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:03:00.639 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:03:00.639 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:03:00.639 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.639 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-08-24T01:03:00.639 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already in. 2020-08-24T01:03:00.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:00.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-08-24T01:03:00.640 INFO:tasks.workunit.client.0.smithi060.stderr:osd.2 is already in. 2020-08-24T01:03:00.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-08-24T01:03:00.641 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-08-24T01:03:00.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-08-24T01:03:00.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-08-24T01:03:00.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:03:00.642 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:03:00.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-08-24T01:03:00.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T01:03:00.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-08-24T01:03:00.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:03:00.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:03:00.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:03:00.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:03:00.643 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T01:03:00.643 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:03:00.644 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:03:00.644 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:03:00.644 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:03:00.644 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:03:00.644 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:03:00.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:03:00.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:03:00.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:03:00.646 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:03:00.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:03:00.646 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:03:00.646 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T01:03:00.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:03:00.647 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.647 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:03:00.647 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888069 2020-08-24T01:03:00.647 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 146028888069 2020-08-24T01:03:00.647 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-146028888069' 2020-08-24T01:03:00.647 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.648 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:03:00.648 INFO:tasks.workunit.client.0.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-08-24T01:03:00.648 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-08-24T01:03:00.648 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z '' 2020-08-24T01:03:00.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: continue 2020-08-24T01:03:00.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:00.649 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:03:00.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855364 2020-08-24T01:03:00.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 150323855364 2020-08-24T01:03:00.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-146028888069 2-150323855364' 2020-08-24T01:03:00.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.650 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-146028888069 2020-08-24T01:03:00.650 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:03:00.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-146028888069 2020-08-24T01:03:00.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=146028888069 2020-08-24T01:03:00.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 146028888069' 2020-08-24T01:03:00.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 146028888067 -lt 146028888069 2020-08-24T01:03:00.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:03:00.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:03:00.652 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 146028888067 -lt 146028888069 2020-08-24T01:03:00.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:03:00.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 299 -eq 0 ']' 2020-08-24T01:03:00.653 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:00.654 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 146028888069 -lt 146028888069 2020-08-24T01:03:00.654 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:00.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-150323855364 2020-08-24T01:03:00.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:00.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:03:00.655 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-150323855364 2020-08-24T01:03:00.655 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:00.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=150323855364 2020-08-24T01:03:00.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 150323855364' 2020-08-24T01:03:00.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:03:00.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 150323855364 -lt 150323855364 2020-08-24T01:03:00.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:03:00.657 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:00.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.659 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.659 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:00.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T01:03:00.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:03:00.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T01:03:00.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:00.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:03:00.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:00.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.663 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:00.663 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.663 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.663 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:00.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:00.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:00.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:00.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:00.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:00.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:00.665 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:00.666 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:00.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:00.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:00.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-08-24T01:03:00.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:00.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-08-24T01:03:00.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:00.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.667 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.667 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.668 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.668 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.668 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.668 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:00.669 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.670 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.670 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.670 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:00.670 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:00.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:00.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:00.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:00.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:00.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:00.672 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:00.672 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:00.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:00.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:00.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-08-24T01:03:00.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:00.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-08-24T01:03:00.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:00.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.673 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.674 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.674 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.674 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.674 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.674 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:00.675 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.675 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.675 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:00.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:00.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:00.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:00.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:00.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:00.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:00.677 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:00.677 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:00.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:00.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:00.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-08-24T01:03:00.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:00.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-08-24T01:03:00.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:00.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:00.681 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.681 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.681 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:00.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:00.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:00.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:00.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:00.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:00.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:00.683 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:00.683 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:00.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:00.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:00.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-08-24T01:03:00.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:00.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-08-24T01:03:00.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:00.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:00.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:00.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:00.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:00.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:00.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:00.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:00.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:00.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:00.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:00.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:00.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:00.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:00.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:00.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:00.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:00.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:00.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:00.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:00.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:00.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:00.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:00.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-08-24T01:03:00.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:00.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-08-24T01:03:00.690 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T01:03:00.690 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:03:00.690 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:03:00.690 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:03:00.691 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/38405,v1:127.0.0.1:6819/38405] [v2:127.0.0.1:6820/38405,v1:127.0.0.1:6821/38405] exists,up 813ed60f-f06b-4b40-a01a-9681d9e7dd5d 2020-08-24T01:03:00.691 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836487 2020-08-24T01:03:00.691 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705669 2020-08-24T01:03:00.691 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542148 2020-08-24T01:03:00.691 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836490 2020-08-24T01:03:00.691 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705672 2020-08-24T01:03:00.692 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542151 2020-08-24T01:03:00.692 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836493 2020-08-24T01:03:00.692 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705675 2020-08-24T01:03:00.692 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542153 2020-08-24T01:03:00.693 INFO:tasks.workunit.client.0.smithi060.stdout:primary and soon to be divergent is 1 2020-08-24T01:03:00.693 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:03:00.693 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:01:37.454593+0000 0'0 24:10 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:03:00.693 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:03:00.693 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:03:00.694 INFO:tasks.workunit.client.0.smithi060.stdout:writing initial objects 2020-08-24T01:03:00.694 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836500 2020-08-24T01:03:00.694 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705682 2020-08-24T01:03:00.694 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542160 2020-08-24T01:03:00.694 INFO:tasks.workunit.client.0.smithi060.stdout:blackholing osds 0 2 2020-08-24T01:03:00.694 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:03:00.695 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 0 0 0 61700 0 0 16 16 active+clean 2020-08-24T01:01:37.454593+0000 27'100 27:110 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:03:00.695 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:03:00.695 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:03:00.695 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:03:00.695 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T01:03:00.696 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:03:00.696 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:03:00.696 INFO:tasks.workunit.client.0.smithi060.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-08-24T01:03:00.696 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:03:00.696 INFO:tasks.workunit.client.0.smithi060.stdout:writing divergent object 2020-08-24T01:03:00.697 INFO:tasks.workunit.client.0.smithi060.stdout:created pool test snap snap1 2020-08-24T01:03:00.697 INFO:tasks.workunit.client.0.smithi060.stdout:killing all the osds 2020-08-24T01:03:00.697 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:03:00.697 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 0 0 0 62317 0 0 14 14 active+clean 2020-08-24T01:01:37.454593+0000 28'104 28:114 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:03:00.697 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:03:00.698 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:03:00.698 INFO:tasks.workunit.client.0.smithi060.stdout:bringing up non_divergent 0 2 2020-08-24T01:03:00.698 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:03:00.698 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 0 0 0 62317 0 0 14 14 stale+active+clean 2020-08-24T01:01:37.454593+0000 28'104 28:114 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:03:00.698 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:03:00.698 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:03:00.699 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:03:00.699 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:03:00.699 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:03:00.699 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/42800,v1:127.0.0.1:6803/42800] [v2:127.0.0.1:6804/42800,v1:127.0.0.1:6805/42800] exists,up 2590757d-b008-4deb-b066-fccba9726114 2020-08-24T01:03:00.699 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:03:00.700 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:03:00.700 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:03:00.700 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,27) [v2:127.0.0.1:6810/42994,v1:127.0.0.1:6811/42994] [v2:127.0.0.1:6812/42994,v1:127.0.0.1:6813/42994] exists,up 813ed60f-f06b-4b40-a01a-9681d9e7dd5d 2020-08-24T01:03:00.700 INFO:tasks.workunit.client.0.smithi060.stdout:writing non-divergent object existing_4 2020-08-24T01:03:00.700 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:03:00.701 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 100 0 100 0 0 61700 0 0 16 16 active+undersized+degraded 2020-08-24T01:02:37.453063+0000 27'100 38:119 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:03:00.701 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:03:00.701 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:03:00.701 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 146028888069 2020-08-24T01:03:00.701 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 150323855364 2020-08-24T01:03:01.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:01.075 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:01.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:01.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:01.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:01.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:01.076 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:01.076 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:01.444 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:01.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:01.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:01.445 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:01.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:01.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:01.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:01.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:01.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:01.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:01.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:01.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:01.900 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:02.347 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:02.348 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:02.348 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-08-24T01:03:02.348 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:02.348 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-08-24T01:03:08.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:08.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:08.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:08.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:08.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:08.752 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:08.752 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:08.752 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:09.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:09.122 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:09.122 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:09.123 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:09.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:09.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:09.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:09.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:09.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:09.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:09.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:09.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:09.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:10.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:10.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:10.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-08-24T01:03:10.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:10.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-08-24T01:03:17.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:17.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:17.338 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:17.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:17.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:17.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:17.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:17.340 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:17.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:17.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:17.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:17.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:18.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:18.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:18.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:18.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:18.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:18.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:18.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:18.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:18.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:18.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:18.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:18.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-08-24T01:03:18.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-08-24T01:03:19.147 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:03:19.148 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_fingerprint": "921035d2-45f2-4241-b709-ae8d4f04bf5a", 2020-08-24T01:03:19.148 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "15.2.4-661-g0a941e5", 2020-08-24T01:03:19.148 INFO:tasks.workunit.client.0.smithi060.stdout: "commit": "0a941e55043ccee6d35a8717bec55ff7581e67d8", 2020-08-24T01:03:19.148 INFO:tasks.workunit.client.0.smithi060.stdout: "timestamp": "2020-08-24T01:03:19.144718+0000", 2020-08-24T01:03:19.148 INFO:tasks.workunit.client.0.smithi060.stdout: "tag": "", 2020-08-24T01:03:19.149 INFO:tasks.workunit.client.0.smithi060.stdout: "health": { 2020-08-24T01:03:19.149 INFO:tasks.workunit.client.0.smithi060.stdout: "status": "HEALTH_WARN", 2020-08-24T01:03:19.149 INFO:tasks.workunit.client.0.smithi060.stdout: "checks": { 2020-08-24T01:03:19.149 INFO:tasks.workunit.client.0.smithi060.stdout: "OSDMAP_FLAGS": { 2020-08-24T01:03:19.154 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:19.155 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:19.155 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "nodown,noout,noin flag(s) set", 2020-08-24T01:03:19.155 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 17 2020-08-24T01:03:19.156 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.156 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [], 2020-08-24T01:03:19.156 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:19.157 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.157 INFO:tasks.workunit.client.0.smithi060.stdout: "OSD_DOWN": { 2020-08-24T01:03:19.157 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:19.157 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:19.158 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 osds down", 2020-08-24T01:03:19.158 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:03:19.158 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.158 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:03:19.158 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.159 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "osd.1 (root=default,host=smithi060) is down" 2020-08-24T01:03:19.159 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.159 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.159 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:19.159 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.160 INFO:tasks.workunit.client.0.smithi060.stdout: "PG_DEGRADED": { 2020-08-24T01:03:19.160 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:19.160 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:19.161 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-08-24T01:03:19.161 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:03:19.161 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.161 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:03:19.162 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.162 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-08-24T01:03:19.162 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.162 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.162 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:19.163 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.163 INFO:tasks.workunit.client.0.smithi060.stdout: "POOL_APP_NOT_ENABLED": { 2020-08-24T01:03:19.163 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:19.163 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:19.164 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 pool(s) do not have an application enabled", 2020-08-24T01:03:19.164 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:03:19.164 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.164 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:03:19.164 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.165 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "application not enabled on pool 'test'" 2020-08-24T01:03:19.165 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.165 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.165 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-08-24T01:03:19.165 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.166 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.169 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:19.169 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.169 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.169 INFO:tasks.workunit.client.0.smithi060.stdout: "mutes": [] 2020-08-24T01:03:19.169 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.170 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_first_committed": 1, 2020-08-24T01:03:19.170 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_last_committed": 1, 2020-08-24T01:03:19.170 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap": { 2020-08-24T01:03:19.170 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:03:19.170 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T01:03:19.171 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T01:01:12.141440Z", 2020-08-24T01:03:19.171 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T01:01:12.141440Z", 2020-08-24T01:03:19.171 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release": 15, 2020-08-24T01:03:19.171 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release_name": "octopus", 2020-08-24T01:03:19.171 INFO:tasks.workunit.client.0.smithi060.stdout: "features": { 2020-08-24T01:03:19.172 INFO:tasks.workunit.client.0.smithi060.stdout: "persistent": [ 2020-08-24T01:03:19.172 INFO:tasks.workunit.client.0.smithi060.stdout: "kraken", 2020-08-24T01:03:19.172 INFO:tasks.workunit.client.0.smithi060.stdout: "luminous", 2020-08-24T01:03:19.172 INFO:tasks.workunit.client.0.smithi060.stdout: "mimic", 2020-08-24T01:03:19.172 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap-prune", 2020-08-24T01:03:19.172 INFO:tasks.workunit.client.0.smithi060.stdout: "nautilus", 2020-08-24T01:03:19.173 INFO:tasks.workunit.client.0.smithi060.stdout: "octopus" 2020-08-24T01:03:19.173 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.173 INFO:tasks.workunit.client.0.smithi060.stdout: "optional": [] 2020-08-24T01:03:19.173 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.173 INFO:tasks.workunit.client.0.smithi060.stdout: "mons": [ 2020-08-24T01:03:19.174 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.179 INFO:tasks.workunit.client.0.smithi060.stdout: "rank": 0, 2020-08-24T01:03:19.179 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "a", 2020-08-24T01:03:19.180 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:19.180 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.180 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.180 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.180 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115", 2020-08-24T01:03:19.181 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 0 2020-08-24T01:03:19.181 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.181 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.181 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.181 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115/0", 2020-08-24T01:03:19.182 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:7115/0", 2020-08-24T01:03:19.182 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 0, 2020-08-24T01:03:19.183 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 0 2020-08-24T01:03:19.183 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.183 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.183 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.183 INFO:tasks.workunit.client.0.smithi060.stdout: "quorum": [ 2020-08-24T01:03:19.184 INFO:tasks.workunit.client.0.smithi060.stdout: 0 2020-08-24T01:03:19.184 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.184 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap": { 2020-08-24T01:03:19.184 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 38, 2020-08-24T01:03:19.184 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T01:03:19.185 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T01:01:12.875133+0000", 2020-08-24T01:03:19.185 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T01:02:37.149944+0000", 2020-08-24T01:03:19.185 INFO:tasks.workunit.client.0.smithi060.stdout: "last_up_change": "2020-08-24T01:02:35.429212+0000", 2020-08-24T01:03:19.185 INFO:tasks.workunit.client.0.smithi060.stdout: "last_in_change": "2020-08-24T01:02:36.827098+0000", 2020-08-24T01:03:19.186 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-08-24T01:03:19.186 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_num": 5800384, 2020-08-24T01:03:19.186 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_set": [ 2020-08-24T01:03:19.186 INFO:tasks.workunit.client.0.smithi060.stdout: "nodown", 2020-08-24T01:03:19.186 INFO:tasks.workunit.client.0.smithi060.stdout: "noin", 2020-08-24T01:03:19.187 INFO:tasks.workunit.client.0.smithi060.stdout: "noout", 2020-08-24T01:03:19.187 INFO:tasks.workunit.client.0.smithi060.stdout: "pglog_hardlimit", 2020-08-24T01:03:19.187 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snapdirs", 2020-08-24T01:03:19.187 INFO:tasks.workunit.client.0.smithi060.stdout: "recovery_deletes", 2020-08-24T01:03:19.187 INFO:tasks.workunit.client.0.smithi060.stdout: "sortbitwise" 2020-08-24T01:03:19.188 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.188 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_version": 8, 2020-08-24T01:03:19.188 INFO:tasks.workunit.client.0.smithi060.stdout: "full_ratio": 0.99000000953674316, 2020-08-24T01:03:19.188 INFO:tasks.workunit.client.0.smithi060.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-08-24T01:03:19.188 INFO:tasks.workunit.client.0.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2020-08-24T01:03:19.189 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_snapshot": "", 2020-08-24T01:03:19.189 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_max": 1, 2020-08-24T01:03:19.189 INFO:tasks.workunit.client.0.smithi060.stdout: "max_osd": 3, 2020-08-24T01:03:19.189 INFO:tasks.workunit.client.0.smithi060.stdout: "require_min_compat_client": "jewel", 2020-08-24T01:03:19.189 INFO:tasks.workunit.client.0.smithi060.stdout: "min_compat_client": "jewel", 2020-08-24T01:03:19.190 INFO:tasks.workunit.client.0.smithi060.stdout: "require_osd_release": "octopus", 2020-08-24T01:03:19.190 INFO:tasks.workunit.client.0.smithi060.stdout: "pools": [ 2020-08-24T01:03:19.190 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.190 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T01:03:19.191 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_name": "test", 2020-08-24T01:03:19.191 INFO:tasks.workunit.client.0.smithi060.stdout: "create_time": "2020-08-24T01:01:36.243978+0000", 2020-08-24T01:03:19.191 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": 16389, 2020-08-24T01:03:19.192 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-08-24T01:03:19.192 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 3, 2020-08-24T01:03:19.193 INFO:tasks.workunit.client.0.smithi060.stdout: "size": 3, 2020-08-24T01:03:19.196 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 2, 2020-08-24T01:03:19.197 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_rule": 1, 2020-08-24T01:03:19.197 INFO:tasks.workunit.client.0.smithi060.stdout: "object_hash": 2, 2020-08-24T01:03:19.197 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_autoscale_mode": "off", 2020-08-24T01:03:19.197 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num": 1, 2020-08-24T01:03:19.197 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num": 1, 2020-08-24T01:03:19.198 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num_target": 1, 2020-08-24T01:03:19.198 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_target": 1, 2020-08-24T01:03:19.198 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_pending": 1, 2020-08-24T01:03:19.198 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pg_merge_meta": { 2020-08-24T01:03:19.198 INFO:tasks.workunit.client.0.smithi060.stdout: "source_pgid": "0.0", 2020-08-24T01:03:19.199 INFO:tasks.workunit.client.0.smithi060.stdout: "ready_epoch": 0, 2020-08-24T01:03:19.199 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_started": 0, 2020-08-24T01:03:19.199 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": 0, 2020-08-24T01:03:19.199 INFO:tasks.workunit.client.0.smithi060.stdout: "source_version": "0'0", 2020-08-24T01:03:19.199 INFO:tasks.workunit.client.0.smithi060.stdout: "target_version": "0'0" 2020-08-24T01:03:19.200 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.200 INFO:tasks.workunit.client.0.smithi060.stdout: "last_change": "28", 2020-08-24T01:03:19.200 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend": "0", 2020-08-24T01:03:19.200 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2020-08-24T01:03:19.200 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2020-08-24T01:03:19.201 INFO:tasks.workunit.client.0.smithi060.stdout: "auid": 0, 2020-08-24T01:03:19.201 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_mode": "pool", 2020-08-24T01:03:19.201 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_seq": 1, 2020-08-24T01:03:19.201 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_epoch": 28, 2020-08-24T01:03:19.202 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_snaps": [ 2020-08-24T01:03:19.202 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.202 INFO:tasks.workunit.client.0.smithi060.stdout: "snapid": 1, 2020-08-24T01:03:19.202 INFO:tasks.workunit.client.0.smithi060.stdout: "stamp": "2020-08-24T01:02:18.063143+0000", 2020-08-24T01:03:19.202 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "snap1" 2020-08-24T01:03:19.203 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.203 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.203 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps": "[]", 2020-08-24T01:03:19.204 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_bytes": 0, 2020-08-24T01:03:19.204 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_objects": 0, 2020-08-24T01:03:19.204 INFO:tasks.workunit.client.0.smithi060.stdout: "tiers": [], 2020-08-24T01:03:19.205 INFO:tasks.workunit.client.0.smithi060.stdout: "tier_of": -1, 2020-08-24T01:03:19.205 INFO:tasks.workunit.client.0.smithi060.stdout: "read_tier": -1, 2020-08-24T01:03:19.205 INFO:tasks.workunit.client.0.smithi060.stdout: "write_tier": -1, 2020-08-24T01:03:19.205 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_mode": "none", 2020-08-24T01:03:19.205 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_bytes": 0, 2020-08-24T01:03:19.206 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_objects": 0, 2020-08-24T01:03:19.206 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-08-24T01:03:19.206 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-08-24T01:03:19.206 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2020-08-24T01:03:19.206 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_flush_age": 0, 2020-08-24T01:03:19.207 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_evict_age": 0, 2020-08-24T01:03:19.207 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profile": "myprofile", 2020-08-24T01:03:19.207 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_params": { 2020-08-24T01:03:19.207 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "none" 2020-08-24T01:03:19.207 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.207 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_period": 0, 2020-08-24T01:03:19.208 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_count": 0, 2020-08-24T01:03:19.208 INFO:tasks.workunit.client.0.smithi060.stdout: "use_gmt_hitset": true, 2020-08-24T01:03:19.208 INFO:tasks.workunit.client.0.smithi060.stdout: "min_read_recency_for_promote": 0, 2020-08-24T01:03:19.208 INFO:tasks.workunit.client.0.smithi060.stdout: "min_write_recency_for_promote": 0, 2020-08-24T01:03:19.208 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2020-08-24T01:03:19.209 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_search_last_n": 0, 2020-08-24T01:03:19.209 INFO:tasks.workunit.client.0.smithi060.stdout: "grade_table": [], 2020-08-24T01:03:19.209 INFO:tasks.workunit.client.0.smithi060.stdout: "stripe_width": 8192, 2020-08-24T01:03:19.209 INFO:tasks.workunit.client.0.smithi060.stdout: "expected_num_objects": 0, 2020-08-24T01:03:19.209 INFO:tasks.workunit.client.0.smithi060.stdout: "fast_read": false, 2020-08-24T01:03:19.210 INFO:tasks.workunit.client.0.smithi060.stdout: "options": {}, 2020-08-24T01:03:19.210 INFO:tasks.workunit.client.0.smithi060.stdout: "application_metadata": {} 2020-08-24T01:03:19.210 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.210 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.210 INFO:tasks.workunit.client.0.smithi060.stdout: "osds": [ 2020-08-24T01:03:19.210 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.211 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:03:19.211 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "2590757d-b008-4deb-b066-fccba9726114", 2020-08-24T01:03:19.211 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T01:03:19.211 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:03:19.211 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:03:19.212 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:03:19.212 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 5, 2020-08-24T01:03:19.212 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 27, 2020-08-24T01:03:19.212 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 34, 2020-08-24T01:03:19.212 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 37, 2020-08-24T01:03:19.213 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 29, 2020-08-24T01:03:19.213 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:03:19.214 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:19.214 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.214 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.214 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.215 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6802", 2020-08-24T01:03:19.215 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.215 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.216 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.216 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.216 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6803", 2020-08-24T01:03:19.217 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.217 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.217 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.217 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.218 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:03:19.218 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.218 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.218 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.219 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6804", 2020-08-24T01:03:19.219 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.219 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.219 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.219 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.220 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6805", 2020-08-24T01:03:19.220 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.220 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.220 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.220 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.221 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:03:19.221 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.221 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.221 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.221 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6808", 2020-08-24T01:03:19.221 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.222 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.222 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.222 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.222 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6809", 2020-08-24T01:03:19.222 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.223 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.223 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.223 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.225 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:03:19.225 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.225 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.225 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.226 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6806", 2020-08-24T01:03:19.226 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.226 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.227 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.227 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.227 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6807", 2020-08-24T01:03:19.227 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42800 2020-08-24T01:03:19.227 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.228 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.228 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.228 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6803/42800", 2020-08-24T01:03:19.228 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6805/42800", 2020-08-24T01:03:19.228 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6809/42800", 2020-08-24T01:03:19.229 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6807/42800", 2020-08-24T01:03:19.229 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:03:19.229 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T01:03:19.229 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T01:03:19.230 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.230 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.230 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.230 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:03:19.230 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "eb457a2e-e0bb-468b-864e-4386438ddbc4", 2020-08-24T01:03:19.231 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 0, 2020-08-24T01:03:19.231 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:03:19.231 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:03:19.231 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:03:19.231 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 0, 2020-08-24T01:03:19.232 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 0, 2020-08-24T01:03:19.232 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 9, 2020-08-24T01:03:19.232 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 23, 2020-08-24T01:03:19.232 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 30, 2020-08-24T01:03:19.232 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:03:19.233 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:19.233 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.233 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.233 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.233 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T01:03:19.234 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.234 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.234 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.235 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.235 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T01:03:19.235 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.235 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.235 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.236 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.236 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:03:19.236 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.236 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.236 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.237 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T01:03:19.237 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.237 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.237 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.237 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.238 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T01:03:19.238 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.238 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.238 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.238 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.239 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:03:19.239 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.239 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.239 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.240 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T01:03:19.240 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.240 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.240 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.240 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.241 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T01:03:19.241 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.241 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.241 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.242 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.242 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:03:19.242 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.243 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.243 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.243 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T01:03:19.243 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.243 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.244 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.244 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.244 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T01:03:19.245 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:19.245 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.245 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.245 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.245 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/38050", 2020-08-24T01:03:19.246 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/38050", 2020-08-24T01:03:19.246 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/38050", 2020-08-24T01:03:19.246 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/38050", 2020-08-24T01:03:19.246 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:03:19.246 INFO:tasks.workunit.client.0.smithi060.stdout: "exists" 2020-08-24T01:03:19.247 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.247 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.247 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.247 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:03:19.247 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "813ed60f-f06b-4b40-a01a-9681d9e7dd5d", 2020-08-24T01:03:19.248 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T01:03:19.248 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:03:19.248 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:03:19.248 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:03:19.248 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 14, 2020-08-24T01:03:19.248 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 27, 2020-08-24T01:03:19.249 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 35, 2020-08-24T01:03:19.249 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 0, 2020-08-24T01:03:19.249 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 31, 2020-08-24T01:03:19.249 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:03:19.249 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:19.250 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.250 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.250 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.250 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T01:03:19.250 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.251 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.251 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.251 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.251 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T01:03:19.251 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.251 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.252 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.252 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.252 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:03:19.252 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.252 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.253 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.253 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T01:03:19.253 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.253 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.253 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.254 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.254 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T01:03:19.254 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.254 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.254 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.255 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.255 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:03:19.256 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.256 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.256 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.256 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T01:03:19.256 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.257 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.257 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.257 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.257 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T01:03:19.257 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.258 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.258 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.258 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.258 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:03:19.258 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:19.259 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.259 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:19.259 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T01:03:19.259 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.260 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.260 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.260 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:19.261 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T01:03:19.261 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 42994 2020-08-24T01:03:19.261 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.261 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.261 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.262 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/42994", 2020-08-24T01:03:19.262 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/42994", 2020-08-24T01:03:19.262 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/42994", 2020-08-24T01:03:19.262 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/42994", 2020-08-24T01:03:19.262 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:03:19.263 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T01:03:19.263 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T01:03:19.263 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.263 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.263 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.263 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_xinfo": [ 2020-08-24T01:03:19.264 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.264 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:03:19.264 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T01:02:29.034749+0000", 2020-08-24T01:03:19.264 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:03:19.264 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:03:19.265 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:03:19.265 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:03:19.265 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T01:01:20.164740+0000", 2020-08-24T01:03:19.266 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:03:19.266 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.266 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.266 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:03:19.266 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T01:02:29.559713+0000", 2020-08-24T01:03:19.267 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:03:19.267 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:03:19.267 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:03:19.267 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:03:19.268 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T01:01:26.672227+0000", 2020-08-24T01:03:19.268 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:03:19.268 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.268 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.269 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:03:19.269 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T01:02:30.078580+0000", 2020-08-24T01:03:19.270 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:03:19.270 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:03:19.270 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:03:19.271 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:03:19.271 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T01:01:33.366933+0000", 2020-08-24T01:03:19.271 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:03:19.271 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.271 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.272 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap": [], 2020-08-24T01:03:19.272 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap_items": [], 2020-08-24T01:03:19.272 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_temp": [], 2020-08-24T01:03:19.272 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_temp": [], 2020-08-24T01:03:19.272 INFO:tasks.workunit.client.0.smithi060.stdout: "blacklist": {}, 2020-08-24T01:03:19.273 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profiles": { 2020-08-24T01:03:19.273 INFO:tasks.workunit.client.0.smithi060.stdout: "default": { 2020-08-24T01:03:19.273 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T01:03:19.273 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "2", 2020-08-24T01:03:19.273 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T01:03:19.274 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van" 2020-08-24T01:03:19.274 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.274 INFO:tasks.workunit.client.0.smithi060.stdout: "myprofile": { 2020-08-24T01:03:19.274 INFO:tasks.workunit.client.0.smithi060.stdout: "crush-device-class": "", 2020-08-24T01:03:19.274 INFO:tasks.workunit.client.0.smithi060.stdout: "crush-failure-domain": "osd", 2020-08-24T01:03:19.275 INFO:tasks.workunit.client.0.smithi060.stdout: "crush-root": "default", 2020-08-24T01:03:19.275 INFO:tasks.workunit.client.0.smithi060.stdout: "jerasure-per-chunk-alignment": "false", 2020-08-24T01:03:19.275 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T01:03:19.275 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "1", 2020-08-24T01:03:19.275 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T01:03:19.275 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van", 2020-08-24T01:03:19.276 INFO:tasks.workunit.client.0.smithi060.stdout: "w": "8" 2020-08-24T01:03:19.276 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.277 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.277 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps_queue": [], 2020-08-24T01:03:19.277 INFO:tasks.workunit.client.0.smithi060.stdout: "new_removed_snaps": [], 2020-08-24T01:03:19.277 INFO:tasks.workunit.client.0.smithi060.stdout: "new_purged_snaps": [], 2020-08-24T01:03:19.277 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_node_flags": {}, 2020-08-24T01:03:19.278 INFO:tasks.workunit.client.0.smithi060.stdout: "device_class_flags": {} 2020-08-24T01:03:19.278 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.278 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_metadata": [ 2020-08-24T01:03:19.278 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.278 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:03:19.279 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:03:19.279 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6804/42800,v1:127.0.0.1:6805/42800]", 2020-08-24T01:03:19.279 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:03:19.279 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:03:19.280 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:03:19.280 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:03:19.280 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:03:19.280 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:03:19.280 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:03:19.281 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:03:19.281 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:03:19.281 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-08-24T01:03:19.281 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:03:19.281 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:03:19.282 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:03:19.282 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:03:19.282 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:03:19.282 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:03:19.282 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:03:19.283 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:03:19.283 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:03:19.283 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:03:19.283 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:03:19.283 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:03:19.284 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:03:19.284 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:03:19.284 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:03:19.284 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6802/42800,v1:127.0.0.1:6803/42800]", 2020-08-24T01:03:19.284 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:03:19.284 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/42800,v1:127.0.0.1:6809/42800]", 2020-08-24T01:03:19.285 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/42800,v1:127.0.0.1:6807/42800]", 2020-08-24T01:03:19.285 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:03:19.285 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:03:19.285 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:03:19.286 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:03:19.286 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:03:19.287 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:03:19.287 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:03:19.287 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:03:19.287 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:03:19.287 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/0", 2020-08-24T01:03:19.288 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:03:19.288 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:03:19.288 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:03:19.288 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.288 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.289 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:03:19.289 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:03:19.289 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/38050,v1:127.0.0.1:6813/38050]", 2020-08-24T01:03:19.289 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:03:19.289 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:03:19.290 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:03:19.290 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:03:19.290 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:03:19.290 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:03:19.290 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:03:19.291 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:03:19.291 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:03:19.291 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-08-24T01:03:19.291 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:03:19.292 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:03:19.292 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:03:19.292 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:03:19.292 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:03:19.292 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:03:19.293 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:03:19.293 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:03:19.293 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:03:19.294 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:03:19.294 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:03:19.294 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:03:19.294 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:03:19.295 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:03:19.295 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:03:19.295 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/38050,v1:127.0.0.1:6811/38050]", 2020-08-24T01:03:19.295 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:03:19.298 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/38050,v1:127.0.0.1:6817/38050]", 2020-08-24T01:03:19.299 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/38050,v1:127.0.0.1:6815/38050]", 2020-08-24T01:03:19.299 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:03:19.299 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:03:19.299 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:03:19.300 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:03:19.300 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:03:19.300 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:03:19.300 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:03:19.300 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:03:19.300 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:03:19.301 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/1", 2020-08-24T01:03:19.301 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:03:19.301 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:03:19.301 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:03:19.302 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.302 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.302 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:03:19.302 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:03:19.302 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/42994,v1:127.0.0.1:6813/42994]", 2020-08-24T01:03:19.303 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:03:19.303 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:03:19.303 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:03:19.303 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:03:19.303 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:03:19.303 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:03:19.304 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:03:19.304 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:03:19.304 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:03:19.304 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-08-24T01:03:19.304 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:03:19.305 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:03:19.305 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:03:19.305 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:03:19.305 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:03:19.305 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:03:19.306 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:03:19.306 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:03:19.306 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:03:19.306 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:03:19.306 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:03:19.306 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:03:19.307 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:03:19.307 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:03:19.307 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:03:19.307 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/42994,v1:127.0.0.1:6811/42994]", 2020-08-24T01:03:19.307 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:03:19.308 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/42994,v1:127.0.0.1:6817/42994]", 2020-08-24T01:03:19.308 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/42994,v1:127.0.0.1:6815/42994]", 2020-08-24T01:03:19.308 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:03:19.308 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:03:19.308 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:03:19.309 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:03:19.309 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:03:19.309 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:03:19.309 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:03:19.309 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:03:19.310 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:03:19.310 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/2", 2020-08-24T01:03:19.310 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:03:19.310 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:03:19.310 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:03:19.311 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.311 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.311 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_clean_epochs": { 2020-08-24T01:03:19.311 INFO:tasks.workunit.client.0.smithi060.stdout: "min_last_epoch_clean": 0, 2020-08-24T01:03:19.311 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": { 2020-08-24T01:03:19.312 INFO:tasks.workunit.client.0.smithi060.stdout: "per_pool": [] 2020-08-24T01:03:19.312 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.312 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_epochs": [ 2020-08-24T01:03:19.312 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.312 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:03:19.312 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 34 2020-08-24T01:03:19.313 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.313 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.313 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:03:19.313 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 9 2020-08-24T01:03:19.313 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.314 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.314 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:03:19.314 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 35 2020-08-24T01:03:19.314 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.314 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.315 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.315 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_first_committed": 1, 2020-08-24T01:03:19.316 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_last_committed": 38, 2020-08-24T01:03:19.316 INFO:tasks.workunit.client.0.smithi060.stdout: "crushmap": { 2020-08-24T01:03:19.316 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": [ 2020-08-24T01:03:19.316 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.317 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:03:19.317 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.0", 2020-08-24T01:03:19.318 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:03:19.318 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.318 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.318 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:03:19.318 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.1", 2020-08-24T01:03:19.319 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:03:19.319 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.319 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.319 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:03:19.319 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.2", 2020-08-24T01:03:19.320 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:03:19.320 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.320 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.320 INFO:tasks.workunit.client.0.smithi060.stdout: "types": [ 2020-08-24T01:03:19.320 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.321 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 0, 2020-08-24T01:03:19.321 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd" 2020-08-24T01:03:19.321 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.321 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.321 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:03:19.322 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "host" 2020-08-24T01:03:19.322 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.322 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.322 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 2, 2020-08-24T01:03:19.322 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "chassis" 2020-08-24T01:03:19.323 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.323 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.323 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 3, 2020-08-24T01:03:19.323 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "rack" 2020-08-24T01:03:19.323 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.324 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.324 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 4, 2020-08-24T01:03:19.324 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "row" 2020-08-24T01:03:19.324 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.324 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.325 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 5, 2020-08-24T01:03:19.325 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pdu" 2020-08-24T01:03:19.325 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.325 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.326 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 6, 2020-08-24T01:03:19.326 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pod" 2020-08-24T01:03:19.326 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.326 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.326 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 7, 2020-08-24T01:03:19.327 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "room" 2020-08-24T01:03:19.327 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.327 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.327 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 8, 2020-08-24T01:03:19.327 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "datacenter" 2020-08-24T01:03:19.327 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.328 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.328 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 9, 2020-08-24T01:03:19.328 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "zone" 2020-08-24T01:03:19.328 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.328 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.329 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 10, 2020-08-24T01:03:19.329 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "region" 2020-08-24T01:03:19.329 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.333 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.333 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:03:19.333 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "root" 2020-08-24T01:03:19.334 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.334 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.334 INFO:tasks.workunit.client.0.smithi060.stdout: "buckets": [ 2020-08-24T01:03:19.334 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.335 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -1, 2020-08-24T01:03:19.335 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default", 2020-08-24T01:03:19.336 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:03:19.336 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T01:03:19.336 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:03:19.336 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:03:19.336 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:03:19.337 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:03:19.337 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.337 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T01:03:19.338 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:03:19.338 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:03:19.338 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.339 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.339 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.339 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.340 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -2, 2020-08-24T01:03:19.340 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default~hdd", 2020-08-24T01:03:19.340 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:03:19.340 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T01:03:19.341 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:03:19.341 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:03:19.341 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:03:19.341 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:03:19.342 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.342 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T01:03:19.342 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:03:19.343 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:03:19.343 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.343 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.344 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.344 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.344 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T01:03:19.345 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060", 2020-08-24T01:03:19.345 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:03:19.345 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T01:03:19.345 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:03:19.346 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:03:19.346 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:03:19.346 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:03:19.347 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.347 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:03:19.347 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:03:19.347 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:03:19.347 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.348 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.348 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:03:19.348 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:03:19.348 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T01:03:19.348 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.348 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.349 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:03:19.349 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:03:19.349 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T01:03:19.349 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.350 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.350 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.350 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.350 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T01:03:19.350 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060~hdd", 2020-08-24T01:03:19.351 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:03:19.351 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T01:03:19.351 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:03:19.351 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:03:19.351 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:03:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:03:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:03:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:03:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:03:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:03:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:03:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T01:03:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:03:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:03:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T01:03:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: "rules": [ 2020-08-24T01:03:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 0, 2020-08-24T01:03:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "replicated_rule", 2020-08-24T01:03:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 0, 2020-08-24T01:03:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T01:03:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 1, 2020-08-24T01:03:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 10, 2020-08-24T01:03:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T01:03:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T01:03:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T01:03:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T01:03:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_firstn", 2020-08-24T01:03:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T01:03:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T01:03:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T01:03:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.360 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.360 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.360 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.360 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 1, 2020-08-24T01:03:19.360 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "test", 2020-08-24T01:03:19.361 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 1, 2020-08-24T01:03:19.361 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 3, 2020-08-24T01:03:19.361 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 3, 2020-08-24T01:03:19.361 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 3, 2020-08-24T01:03:19.361 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T01:03:19.362 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.362 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "set_chooseleaf_tries", 2020-08-24T01:03:19.362 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 5 2020-08-24T01:03:19.362 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.362 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.362 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "set_choose_tries", 2020-08-24T01:03:19.363 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 100 2020-08-24T01:03:19.363 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.363 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.363 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T01:03:19.363 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T01:03:19.364 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T01:03:19.364 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.364 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.364 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_indep", 2020-08-24T01:03:19.364 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T01:03:19.365 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T01:03:19.365 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.365 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.365 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T01:03:19.365 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.365 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:19.366 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.366 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.366 INFO:tasks.workunit.client.0.smithi060.stdout: "tunables": { 2020-08-24T01:03:19.366 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_tries": 0, 2020-08-24T01:03:19.366 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_fallback_tries": 0, 2020-08-24T01:03:19.367 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_total_tries": 50, 2020-08-24T01:03:19.367 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_descend_once": 1, 2020-08-24T01:03:19.367 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_vary_r": 1, 2020-08-24T01:03:19.367 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_stable": 1, 2020-08-24T01:03:19.367 INFO:tasks.workunit.client.0.smithi060.stdout: "straw_calc_version": 1, 2020-08-24T01:03:19.368 INFO:tasks.workunit.client.0.smithi060.stdout: "allowed_bucket_algs": 54, 2020-08-24T01:03:19.368 INFO:tasks.workunit.client.0.smithi060.stdout: "profile": "jewel", 2020-08-24T01:03:19.368 INFO:tasks.workunit.client.0.smithi060.stdout: "optimal_tunables": 1, 2020-08-24T01:03:19.368 INFO:tasks.workunit.client.0.smithi060.stdout: "legacy_tunables": 0, 2020-08-24T01:03:19.368 INFO:tasks.workunit.client.0.smithi060.stdout: "minimum_required_version": "jewel", 2020-08-24T01:03:19.369 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables": 1, 2020-08-24T01:03:19.369 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables2": 1, 2020-08-24T01:03:19.369 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v2_rules": 1, 2020-08-24T01:03:19.369 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables3": 1, 2020-08-24T01:03:19.369 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v3_rules": 0, 2020-08-24T01:03:19.370 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v4_buckets": 1, 2020-08-24T01:03:19.370 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables5": 1, 2020-08-24T01:03:19.370 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v5_rules": 0 2020-08-24T01:03:19.370 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.370 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_args": {} 2020-08-24T01:03:19.371 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.371 INFO:tasks.workunit.client.0.smithi060.stdout: "fsmap": { 2020-08-24T01:03:19.371 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:03:19.371 INFO:tasks.workunit.client.0.smithi060.stdout: "default_fscid": -1, 2020-08-24T01:03:19.371 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": { 2020-08-24T01:03:19.372 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": {}, 2020-08-24T01:03:19.372 INFO:tasks.workunit.client.0.smithi060.stdout: "ro_compat": {}, 2020-08-24T01:03:19.372 INFO:tasks.workunit.client.0.smithi060.stdout: "incompat": { 2020-08-24T01:03:19.373 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_1": "base v0.20", 2020-08-24T01:03:19.373 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_2": "client writeable ranges", 2020-08-24T01:03:19.373 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_3": "default file layouts on dirs", 2020-08-24T01:03:19.373 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_4": "dir inode in separate object", 2020-08-24T01:03:19.373 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_5": "mds uses versioned encoding", 2020-08-24T01:03:19.374 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_6": "dirfrag is stored in omap", 2020-08-24T01:03:19.374 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_8": "no anchor table", 2020-08-24T01:03:19.374 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_9": "file layout v2", 2020-08-24T01:03:19.374 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_10": "snaprealm v2" 2020-08-24T01:03:19.375 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.375 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.375 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_flags": { 2020-08-24T01:03:19.375 INFO:tasks.workunit.client.0.smithi060.stdout: "enable_multiple": false, 2020-08-24T01:03:19.375 INFO:tasks.workunit.client.0.smithi060.stdout: "ever_enabled_multiple": false 2020-08-24T01:03:19.376 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.376 INFO:tasks.workunit.client.0.smithi060.stdout: "standbys": [], 2020-08-24T01:03:19.376 INFO:tasks.workunit.client.0.smithi060.stdout: "filesystems": [] 2020-08-24T01:03:19.376 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.377 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_first_committed": 1, 2020-08-24T01:03:19.377 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_last_committed": 1, 2020-08-24T01:03:19.378 INFO:tasks.workunit.client.0.smithi060.stdout: "auth": { 2020-08-24T01:03:19.378 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T01:03:19.378 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 5, 2020-08-24T01:03:19.378 INFO:tasks.workunit.client.0.smithi060.stdout: "num_secrets": 3 2020-08-24T01:03:19.378 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.378 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T01:03:19.379 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_active": 1, 2020-08-24T01:03:19.379 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_unknown": 0, 2020-08-24T01:03:19.379 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osd": 3, 2020-08-24T01:03:19.379 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_sum": { 2020-08-24T01:03:19.379 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T01:03:19.380 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 62471, 2020-08-24T01:03:19.380 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 101, 2020-08-24T01:03:19.380 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 1, 2020-08-24T01:03:19.380 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 303, 2020-08-24T01:03:19.380 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T01:03:19.381 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T01:03:19.381 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 101, 2020-08-24T01:03:19.381 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T01:03:19.381 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T01:03:19.381 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 101, 2020-08-24T01:03:19.382 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T01:03:19.382 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T01:03:19.382 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T01:03:19.382 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T01:03:19.382 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T01:03:19.382 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T01:03:19.383 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T01:03:19.383 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T01:03:19.383 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T01:03:19.383 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T01:03:19.383 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T01:03:19.384 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T01:03:19.384 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T01:03:19.384 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T01:03:19.384 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T01:03:19.384 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T01:03:19.384 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T01:03:19.385 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T01:03:19.385 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T01:03:19.385 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T01:03:19.385 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T01:03:19.386 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T01:03:19.386 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T01:03:19.386 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T01:03:19.386 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T01:03:19.386 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T01:03:19.386 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T01:03:19.387 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T01:03:19.387 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T01:03:19.387 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T01:03:19.387 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.387 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T01:03:19.388 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T01:03:19.388 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T01:03:19.388 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T01:03:19.388 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 0, 2020-08-24T01:03:19.388 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 0, 2020-08-24T01:03:19.389 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:03:19.389 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:03:19.389 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:03:19.389 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T01:03:19.389 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T01:03:19.390 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.390 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 18, 2020-08-24T01:03:19.390 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 18, 2020-08-24T01:03:19.390 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 3, 2020-08-24T01:03:19.390 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 3, 2020-08-24T01:03:19.391 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 0 2020-08-24T01:03:19.391 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.391 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum": { 2020-08-24T01:03:19.391 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T01:03:19.391 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T01:03:19.392 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T01:03:19.392 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T01:03:19.393 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T01:03:19.393 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T01:03:19.393 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T01:03:19.393 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3179136, 2020-08-24T01:03:19.394 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 33088, 2020-08-24T01:03:19.394 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 10, 2020-08-24T01:03:19.394 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145717, 2020-08-24T01:03:19.395 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311393664, 2020-08-24T01:03:19.395 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T01:03:19.395 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T01:03:19.395 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318867111936, 2020-08-24T01:03:19.395 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T01:03:19.396 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 33882112, 2020-08-24T01:03:19.396 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 1512225, 2020-08-24T01:03:19.396 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:03:19.396 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:03:19.396 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:03:19.396 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 10482, 2020-08-24T01:03:19.397 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221214990 2020-08-24T01:03:19.397 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.397 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T01:03:19.397 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T01:03:19.397 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T01:03:19.398 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T01:03:19.398 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T01:03:19.398 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T01:03:19.398 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T01:03:19.398 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.399 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T01:03:19.399 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T01:03:19.399 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T01:03:19.399 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T01:03:19.399 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T01:03:19.400 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.400 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T01:03:19.400 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T01:03:19.400 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.400 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum_by_class": { 2020-08-24T01:03:19.401 INFO:tasks.workunit.client.0.smithi060.stdout: "hdd": { 2020-08-24T01:03:19.401 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T01:03:19.401 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T01:03:19.401 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T01:03:19.401 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T01:03:19.402 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T01:03:19.402 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T01:03:19.402 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T01:03:19.402 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3179136, 2020-08-24T01:03:19.403 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 33088, 2020-08-24T01:03:19.403 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 10, 2020-08-24T01:03:19.403 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145717, 2020-08-24T01:03:19.403 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311393664, 2020-08-24T01:03:19.403 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T01:03:19.404 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T01:03:19.404 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318867111936, 2020-08-24T01:03:19.404 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T01:03:19.404 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 33882112, 2020-08-24T01:03:19.404 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 1512225, 2020-08-24T01:03:19.404 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:03:19.405 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:03:19.405 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:03:19.405 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 10482, 2020-08-24T01:03:19.405 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221214990 2020-08-24T01:03:19.405 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.406 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T01:03:19.406 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T01:03:19.406 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T01:03:19.406 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T01:03:19.406 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T01:03:19.407 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T01:03:19.407 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T01:03:19.407 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.407 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T01:03:19.407 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T01:03:19.407 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T01:03:19.408 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T01:03:19.408 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T01:03:19.408 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.408 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T01:03:19.408 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T01:03:19.409 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.409 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.409 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_stats": [ 2020-08-24T01:03:19.409 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.409 INFO:tasks.workunit.client.0.smithi060.stdout: "poolid": 1, 2020-08-24T01:03:19.410 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T01:03:19.410 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T01:03:19.410 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 62471, 2020-08-24T01:03:19.410 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 101, 2020-08-24T01:03:19.410 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 1, 2020-08-24T01:03:19.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 303, 2020-08-24T01:03:19.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T01:03:19.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T01:03:19.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 101, 2020-08-24T01:03:19.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T01:03:19.411 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T01:03:19.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 101, 2020-08-24T01:03:19.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T01:03:19.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T01:03:19.412 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T01:03:19.413 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T01:03:19.413 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T01:03:19.413 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T01:03:19.413 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T01:03:19.414 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T01:03:19.414 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T01:03:19.414 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T01:03:19.414 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T01:03:19.415 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T01:03:19.415 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T01:03:19.415 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T01:03:19.415 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T01:03:19.415 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T01:03:19.415 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T01:03:19.416 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T01:03:19.416 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T01:03:19.416 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T01:03:19.416 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T01:03:19.416 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T01:03:19.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T01:03:19.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T01:03:19.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T01:03:19.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T01:03:19.417 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T01:03:19.418 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T01:03:19.418 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T01:03:19.418 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T01:03:19.418 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.418 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T01:03:19.419 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T01:03:19.419 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T01:03:19.419 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T01:03:19.419 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 19857408, 2020-08-24T01:03:19.419 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 1241088, 2020-08-24T01:03:19.419 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:03:19.420 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:03:19.420 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:03:19.420 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T01:03:19.420 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T01:03:19.420 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.421 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 18, 2020-08-24T01:03:19.421 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 18, 2020-08-24T01:03:19.421 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 3, 2020-08-24T01:03:19.421 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 3, 2020-08-24T01:03:19.421 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 3 2020-08-24T01:03:19.421 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.422 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.422 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_stats": [ 2020-08-24T01:03:19.422 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.422 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:03:19.422 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 146028888075 2020-08-24T01:03:19.423 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.423 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.423 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:03:19.423 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 38654705685 2020-08-24T01:03:19.424 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.424 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.424 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:03:19.424 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 150323855371 2020-08-24T01:03:19.424 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.425 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.425 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_state": [ 2020-08-24T01:03:19.425 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.425 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "active+undersized+degraded", 2020-08-24T01:03:19.425 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 1 2020-08-24T01:03:19.426 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.426 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.426 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_osd": [ 2020-08-24T01:03:19.426 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.426 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:03:19.427 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:03:19.427 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T01:03:19.427 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:03:19.427 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.428 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.428 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2147483647, 2020-08-24T01:03:19.428 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:03:19.428 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T01:03:19.429 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:03:19.429 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.429 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.429 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:03:19.429 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:03:19.429 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 0, 2020-08-24T01:03:19.430 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:03:19.430 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.430 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.431 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:03:19.431 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 1, 2020-08-24T01:03:19.431 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T01:03:19.431 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:03:19.431 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.432 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.432 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": [ 2020-08-24T01:03:19.432 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:19.432 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T01:03:19.432 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": {} 2020-08-24T01:03:19.432 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.433 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:19.433 INFO:tasks.workunit.client.0.smithi060.stdout: "servicemap": { 2020-08-24T01:03:19.433 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:03:19.433 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T01:01:12.875742+0000", 2020-08-24T01:03:19.434 INFO:tasks.workunit.client.0.smithi060.stdout: "services": {} 2020-08-24T01:03:19.434 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:19.434 INFO:tasks.workunit.client.0.smithi060.stdout: "paxos": { 2020-08-24T01:03:19.434 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T01:03:19.435 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 385, 2020-08-24T01:03:19.435 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pn": 0, 2020-08-24T01:03:19.435 INFO:tasks.workunit.client.0.smithi060.stdout: "accepted_pn": 0 2020-08-24T01:03:19.435 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:19.435 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:03:19.437 INFO:tasks.workunit.client.0.smithi060.stderr:report 2568655465 2020-08-24T01:03:19.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-08-24T01:03:19.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:19.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-08-24T01:03:19.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:03:19.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:03:19.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:03:19.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:03:19.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:03:19.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:03:19.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-08-24T01:03:19.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-08-24T01:03:19.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:03:19.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-08-24T01:03:19.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:03:19.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-08-24T01:03:19.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-08-24T01:03:19.927 INFO:tasks.workunit.client.0.smithi060.stderr:PG '1.0' not found 2020-08-24T01:03:20.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-08-24T01:03:20.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-08-24T01:03:20.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:03:20.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:03:20.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-08-24T01:03:20.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:03:20.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-08-24T01:03:20.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:03:20.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:03:20.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:03:20.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:03:20.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-08-24T01:03:20.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-08-24T01:03:20.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:03:20.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:03:20.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:03:20.467 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:03:20.467 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:03:20.467 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:03:20.467 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:03:20.468 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:03:20.468 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:03:20.470 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:20.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:20.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:03:20.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:03:20.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:03:20.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:03:20.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:03:20.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:20.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:03:20.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:03:20.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-08-24T01:03:20.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-08-24T01:03:20.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:20.485 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-08-24T01:03:20.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-08-24T01:03:20.485 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:03:20.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:03:20.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:03:20.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:03:20.505 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:20.501+0000 7fe067e59d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:20.508 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:20.505+0000 7fe067e59d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:20.510 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:20.505+0000 7fe067e59d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:20.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-08-24T01:03:20.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:03:20.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:03:20.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:03:20.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:03:20.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:20.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:03:20.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:20.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:03:20.846 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:03:21.049 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:21.045+0000 7fe067e59d80 -1 Falling back to public interface 2020-08-24T01:03:21.195 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 34 up_thru 37 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/42800,v1:127.0.0.1:6803/42800] [v2:127.0.0.1:6804/42800,v1:127.0.0.1:6805/42800] exists,up 2590757d-b008-4deb-b066-fccba9726114 2020-08-24T01:03:21.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:03:21.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:03:21.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:03:21.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:03:21.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-08-24T01:03:21.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:03:21.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:03:21.198 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:03:21.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:03:21.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:03:21.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:03:21.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-08-24T01:03:21.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-08-24T01:03:21.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:03:21.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-08-24T01:03:21.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:03:21.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-08-24T01:03:21.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-08-24T01:03:22.129 INFO:tasks.workunit.client.0.smithi060.stderr:PG '1.0' not found 2020-08-24T01:03:22.164 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:22.161+0000 7fe067e59d80 -1 osd.0 38 log_to_monitors {default=true} 2020-08-24T01:03:22.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-08-24T01:03:22.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-08-24T01:03:22.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:03:22.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:03:22.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-08-24T01:03:22.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:03:22.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-08-24T01:03:22.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:03:22.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:03:22.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:03:22.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:03:22.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-08-24T01:03:22.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-08-24T01:03:22.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:03:22.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:03:22.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:03:22.672 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:03:22.672 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:03:22.673 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:03:22.673 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:03:22.673 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:03:22.673 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:03:22.674 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:22.674 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:03:22.674 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:03:22.674 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:03:22.674 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:03:22.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:03:22.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:03:22.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:22.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:03:22.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:03:22.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-08-24T01:03:22.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-08-24T01:03:22.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:03:22.676 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-08-24T01:03:22.677 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:03:22.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-08-24T01:03:22.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:03:22.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:03:22.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:03:22.710 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:22.705+0000 7fb28a432d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:22.718 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:22.709+0000 7fb28a432d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:22.718 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:22.709+0000 7fb28a432d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:03:23.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-08-24T01:03:23.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:03:23.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:03:23.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:03:23.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:03:23.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:03:23.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:03:23.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:03:23.058 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:03:23.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:03:23.261 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:23.257+0000 7fb28a432d80 -1 Falling back to public interface 2020-08-24T01:03:23.463 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,27) [v2:127.0.0.1:6810/42994,v1:127.0.0.1:6811/42994] [v2:127.0.0.1:6812/42994,v1:127.0.0.1:6813/42994] exists,up 813ed60f-f06b-4b40-a01a-9681d9e7dd5d 2020-08-24T01:03:23.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:03:23.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:03:23.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:03:23.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-08-24T01:03:23.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-08-24T01:03:23.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:03:23.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-08-24T01:03:23.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:03:23.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-08-24T01:03:23.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-08-24T01:03:24.303 INFO:tasks.workunit.client.0.smithi060.stderr:PG '1.0' not found 2020-08-24T01:03:24.310 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:03:24.305+0000 7fb28a432d80 -1 osd.2 38 log_to_monitors {default=true} 2020-08-24T01:03:24.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-08-24T01:03:24.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-08-24T01:03:24.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-08-24T01:03:24.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:03:24.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:03:24.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:03:24.840 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:03:24.841 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-08-24T01:03:24.841 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:03:24.842 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:03:24.842 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:03:24.842 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:03:24.843 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:03:24.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:03:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:03:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:03:24.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:03:24.977 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:03:25.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:03:25.393 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:03:25.394 INFO:tasks.workunit.client.0.smithi060.stderr:2' 2020-08-24T01:03:25.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:03:25.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:25.395 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:03:25.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=171798691843 2020-08-24T01:03:25.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 171798691843 2020-08-24T01:03:25.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-171798691843' 2020-08-24T01:03:25.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:25.564 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:03:25.689 INFO:tasks.workunit.client.0.smithi060.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-08-24T01:03:25.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-08-24T01:03:25.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z '' 2020-08-24T01:03:25.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2074: flush_pg_stats: continue 2020-08-24T01:03:25.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:03:25.690 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:03:25.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=184683593731 2020-08-24T01:03:25.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 184683593731 2020-08-24T01:03:25.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-171798691843 2-184683593731' 2020-08-24T01:03:25.849 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:25.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-171798691843 2020-08-24T01:03:25.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:25.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:03:25.851 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-171798691843 2020-08-24T01:03:25.851 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:25.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=171798691843 2020-08-24T01:03:25.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 171798691843' 2020-08-24T01:03:25.854 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:25.854 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 171798691843 2020-08-24T01:03:26.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 1 -lt 171798691843 2020-08-24T01:03:26.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:03:27.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:03:27.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:03:27.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 171798691844 -lt 171798691843 2020-08-24T01:03:27.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:03:27.586 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-184683593731 2020-08-24T01:03:27.586 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:03:27.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:03:27.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:03:27.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-184683593731 2020-08-24T01:03:27.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=184683593731 2020-08-24T01:03:27.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2020-08-24T01:03:27.593 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:03:27.593 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 184683593731 2020-08-24T01:03:27.962 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 184683593731 -lt 184683593731 2020-08-24T01:03:27.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:03:27.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:27.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:28.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:03:28.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:28.401 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:28.401 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:28.401 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:28.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:28.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:28.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:28.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:28.768 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:28.768 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:28.769 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:29.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:29.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T01:03:29.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:03:29.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T01:03:29.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:29.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:03:29.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:29.336 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:29.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:29.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:29.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:29.348 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:29.349 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:29.349 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:29.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:29.691 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:29.691 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:29.691 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:30.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:30.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:30.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:30.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:30.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:30.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:30.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:30.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:30.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:30.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:30.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:30.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-08-24T01:03:30.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:30.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-08-24T01:03:30.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:30.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:30.792 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:30.793 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:30.793 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:30.794 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:30.794 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:30.794 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:31.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:31.142 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:31.142 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:31.142 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:31.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:31.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:31.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:31.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:31.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:31.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:31.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:31.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:31.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:32.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:32.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:32.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-08-24T01:03:32.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:32.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-08-24T01:03:32.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:32.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:32.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:32.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:32.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:32.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:32.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:32.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:32.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:32.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:32.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:32.818 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:33.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:33.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:33.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:33.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:33.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:33.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:33.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:33.270 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:33.270 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:33.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:33.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:33.700 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-08-24T01:03:33.700 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:33.700 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-08-24T01:03:34.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:34.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:34.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:34.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:34.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:34.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:34.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:34.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:34.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:34.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:34.876 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:34.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:35.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:49.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:49.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:49.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:49.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:49.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:49.751 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:49.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:49.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:49.757 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:49.758 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:49.759 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-08-24T01:03:49.760 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:49.760 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-08-24T01:03:49.760 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:49.760 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:49.761 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:49.761 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:49.761 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:49.761 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:49.761 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:49.762 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:49.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:49.762 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:49.762 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:49.762 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:49.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:49.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:49.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:49.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:49.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:49.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:49.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:49.764 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:49.764 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:49.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:49.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:49.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-08-24T01:03:49.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:49.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-08-24T01:03:49.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:49.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:49.766 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:49.766 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:49.766 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:49.767 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:49.767 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:49.767 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:49.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:49.768 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:49.768 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:49.768 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:49.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:49.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:49.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:49.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:49.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:49.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:49.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:49.770 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:49.770 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:49.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:49.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:49.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-08-24T01:03:49.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:49.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-08-24T01:03:49.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:49.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:49.771 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:49.772 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:49.772 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:49.772 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:49.772 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:49.772 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:49.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:49.927 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:49.928 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:49.928 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:50.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:50.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:50.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:50.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:50.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:50.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:50.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:50.385 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:50.385 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:50.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:50.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:50.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-08-24T01:03:50.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:03:50.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-08-24T01:03:58.125 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:03:58.125 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:03:58.126 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:03:58.126 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:03:58.126 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:03:58.127 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:03:58.127 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:03:58.127 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:03:58.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:03:58.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:03:58.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:03:58.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:03:58.958 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:03:58.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:03:58.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:03:58.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:03:58.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:03:58.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:03:58.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:03:58.960 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:03:58.961 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:03:59.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-08-24T01:03:59.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-08-24T01:03:59.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-08-24T01:03:59.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-08-24T01:03:59.862 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:03:59.862 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_fingerprint": "921035d2-45f2-4241-b709-ae8d4f04bf5a", 2020-08-24T01:03:59.863 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "15.2.4-661-g0a941e5", 2020-08-24T01:03:59.863 INFO:tasks.workunit.client.0.smithi060.stdout: "commit": "0a941e55043ccee6d35a8717bec55ff7581e67d8", 2020-08-24T01:03:59.863 INFO:tasks.workunit.client.0.smithi060.stdout: "timestamp": "2020-08-24T01:03:59.859047+0000", 2020-08-24T01:03:59.863 INFO:tasks.workunit.client.0.smithi060.stdout: "tag": "", 2020-08-24T01:03:59.863 INFO:tasks.workunit.client.0.smithi060.stdout: "health": { 2020-08-24T01:03:59.864 INFO:tasks.workunit.client.0.smithi060.stdout: "status": "HEALTH_WARN", 2020-08-24T01:03:59.864 INFO:tasks.workunit.client.0.smithi060.stdout: "checks": { 2020-08-24T01:03:59.864 INFO:tasks.workunit.client.0.smithi060.stdout: "OSDMAP_FLAGS": { 2020-08-24T01:03:59.864 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:59.864 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:59.865 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "nodown,noout,noin flag(s) set", 2020-08-24T01:03:59.865 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 17 2020-08-24T01:03:59.865 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.865 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [], 2020-08-24T01:03:59.865 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:59.866 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.866 INFO:tasks.workunit.client.0.smithi060.stdout: "OSD_DOWN": { 2020-08-24T01:03:59.866 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:59.866 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:59.866 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 osds down", 2020-08-24T01:03:59.867 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:03:59.867 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.867 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:03:59.867 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.867 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "osd.1 (root=default,host=smithi060) is down" 2020-08-24T01:03:59.868 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.868 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.868 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:59.868 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.868 INFO:tasks.workunit.client.0.smithi060.stdout: "PG_DEGRADED": { 2020-08-24T01:03:59.868 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:59.869 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:59.869 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-08-24T01:03:59.869 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:03:59.869 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.869 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:03:59.870 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.870 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-08-24T01:03:59.870 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.870 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.870 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:59.871 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.871 INFO:tasks.workunit.client.0.smithi060.stdout: "POOL_APP_NOT_ENABLED": { 2020-08-24T01:03:59.871 INFO:tasks.workunit.client.0.smithi060.stdout: "severity": "HEALTH_WARN", 2020-08-24T01:03:59.871 INFO:tasks.workunit.client.0.smithi060.stdout: "summary": { 2020-08-24T01:03:59.871 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "1 pool(s) do not have an application enabled", 2020-08-24T01:03:59.871 INFO:tasks.workunit.client.0.smithi060.stdout: "count": 1 2020-08-24T01:03:59.872 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.872 INFO:tasks.workunit.client.0.smithi060.stdout: "detail": [ 2020-08-24T01:03:59.872 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.872 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "application not enabled on pool 'test'" 2020-08-24T01:03:59.872 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.873 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.873 INFO:tasks.workunit.client.0.smithi060.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-08-24T01:03:59.873 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.873 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.873 INFO:tasks.workunit.client.0.smithi060.stdout: "muted": false 2020-08-24T01:03:59.874 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.874 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.874 INFO:tasks.workunit.client.0.smithi060.stdout: "mutes": [] 2020-08-24T01:03:59.874 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.874 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_first_committed": 1, 2020-08-24T01:03:59.874 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap_last_committed": 1, 2020-08-24T01:03:59.875 INFO:tasks.workunit.client.0.smithi060.stdout: "monmap": { 2020-08-24T01:03:59.875 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:03:59.875 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T01:03:59.875 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T01:01:12.141440Z", 2020-08-24T01:03:59.875 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T01:01:12.141440Z", 2020-08-24T01:03:59.876 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release": 15, 2020-08-24T01:03:59.876 INFO:tasks.workunit.client.0.smithi060.stdout: "min_mon_release_name": "octopus", 2020-08-24T01:03:59.876 INFO:tasks.workunit.client.0.smithi060.stdout: "features": { 2020-08-24T01:03:59.876 INFO:tasks.workunit.client.0.smithi060.stdout: "persistent": [ 2020-08-24T01:03:59.876 INFO:tasks.workunit.client.0.smithi060.stdout: "kraken", 2020-08-24T01:03:59.877 INFO:tasks.workunit.client.0.smithi060.stdout: "luminous", 2020-08-24T01:03:59.877 INFO:tasks.workunit.client.0.smithi060.stdout: "mimic", 2020-08-24T01:03:59.877 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap-prune", 2020-08-24T01:03:59.877 INFO:tasks.workunit.client.0.smithi060.stdout: "nautilus", 2020-08-24T01:03:59.877 INFO:tasks.workunit.client.0.smithi060.stdout: "octopus" 2020-08-24T01:03:59.878 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.878 INFO:tasks.workunit.client.0.smithi060.stdout: "optional": [] 2020-08-24T01:03:59.878 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.878 INFO:tasks.workunit.client.0.smithi060.stdout: "mons": [ 2020-08-24T01:03:59.878 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.879 INFO:tasks.workunit.client.0.smithi060.stdout: "rank": 0, 2020-08-24T01:03:59.879 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "a", 2020-08-24T01:03:59.879 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:59.879 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.879 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.879 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.880 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115", 2020-08-24T01:03:59.880 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 0 2020-08-24T01:03:59.880 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.880 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.880 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.881 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:7115/0", 2020-08-24T01:03:59.881 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:7115/0", 2020-08-24T01:03:59.881 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 0, 2020-08-24T01:03:59.881 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 0 2020-08-24T01:03:59.881 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.882 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.882 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.882 INFO:tasks.workunit.client.0.smithi060.stdout: "quorum": [ 2020-08-24T01:03:59.882 INFO:tasks.workunit.client.0.smithi060.stdout: 0 2020-08-24T01:03:59.882 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.883 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap": { 2020-08-24T01:03:59.883 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 44, 2020-08-24T01:03:59.883 INFO:tasks.workunit.client.0.smithi060.stdout: "fsid": "a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0", 2020-08-24T01:03:59.883 INFO:tasks.workunit.client.0.smithi060.stdout: "created": "2020-08-24T01:01:12.875133+0000", 2020-08-24T01:03:59.883 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T01:03:24.711963+0000", 2020-08-24T01:03:59.883 INFO:tasks.workunit.client.0.smithi060.stdout: "last_up_change": "2020-08-24T01:03:24.561169+0000", 2020-08-24T01:03:59.884 INFO:tasks.workunit.client.0.smithi060.stdout: "last_in_change": "2020-08-24T01:02:36.827098+0000", 2020-08-24T01:03:59.884 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-08-24T01:03:59.884 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_num": 5800384, 2020-08-24T01:03:59.884 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_set": [ 2020-08-24T01:03:59.884 INFO:tasks.workunit.client.0.smithi060.stdout: "nodown", 2020-08-24T01:03:59.885 INFO:tasks.workunit.client.0.smithi060.stdout: "noin", 2020-08-24T01:03:59.885 INFO:tasks.workunit.client.0.smithi060.stdout: "noout", 2020-08-24T01:03:59.885 INFO:tasks.workunit.client.0.smithi060.stdout: "pglog_hardlimit", 2020-08-24T01:03:59.885 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snapdirs", 2020-08-24T01:03:59.885 INFO:tasks.workunit.client.0.smithi060.stdout: "recovery_deletes", 2020-08-24T01:03:59.886 INFO:tasks.workunit.client.0.smithi060.stdout: "sortbitwise" 2020-08-24T01:03:59.886 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.886 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_version": 8, 2020-08-24T01:03:59.886 INFO:tasks.workunit.client.0.smithi060.stdout: "full_ratio": 0.99000000953674316, 2020-08-24T01:03:59.886 INFO:tasks.workunit.client.0.smithi060.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-08-24T01:03:59.886 INFO:tasks.workunit.client.0.smithi060.stdout: "nearfull_ratio": 0.85000002384185791, 2020-08-24T01:03:59.887 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_snapshot": "", 2020-08-24T01:03:59.887 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_max": 1, 2020-08-24T01:03:59.887 INFO:tasks.workunit.client.0.smithi060.stdout: "max_osd": 3, 2020-08-24T01:03:59.887 INFO:tasks.workunit.client.0.smithi060.stdout: "require_min_compat_client": "jewel", 2020-08-24T01:03:59.887 INFO:tasks.workunit.client.0.smithi060.stdout: "min_compat_client": "jewel", 2020-08-24T01:03:59.888 INFO:tasks.workunit.client.0.smithi060.stdout: "require_osd_release": "octopus", 2020-08-24T01:03:59.888 INFO:tasks.workunit.client.0.smithi060.stdout: "pools": [ 2020-08-24T01:03:59.888 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.888 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T01:03:59.888 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_name": "test", 2020-08-24T01:03:59.889 INFO:tasks.workunit.client.0.smithi060.stdout: "create_time": "2020-08-24T01:01:36.243978+0000", 2020-08-24T01:03:59.889 INFO:tasks.workunit.client.0.smithi060.stdout: "flags": 16389, 2020-08-24T01:03:59.889 INFO:tasks.workunit.client.0.smithi060.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-08-24T01:03:59.889 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 3, 2020-08-24T01:03:59.889 INFO:tasks.workunit.client.0.smithi060.stdout: "size": 3, 2020-08-24T01:03:59.889 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 2, 2020-08-24T01:03:59.890 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_rule": 1, 2020-08-24T01:03:59.890 INFO:tasks.workunit.client.0.smithi060.stdout: "object_hash": 2, 2020-08-24T01:03:59.890 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_autoscale_mode": "off", 2020-08-24T01:03:59.890 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num": 1, 2020-08-24T01:03:59.890 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num": 1, 2020-08-24T01:03:59.891 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_placement_num_target": 1, 2020-08-24T01:03:59.891 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_target": 1, 2020-08-24T01:03:59.891 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_num_pending": 1, 2020-08-24T01:03:59.891 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pg_merge_meta": { 2020-08-24T01:03:59.891 INFO:tasks.workunit.client.0.smithi060.stdout: "source_pgid": "0.0", 2020-08-24T01:03:59.892 INFO:tasks.workunit.client.0.smithi060.stdout: "ready_epoch": 0, 2020-08-24T01:03:59.892 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_started": 0, 2020-08-24T01:03:59.892 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": 0, 2020-08-24T01:03:59.892 INFO:tasks.workunit.client.0.smithi060.stdout: "source_version": "0'0", 2020-08-24T01:03:59.892 INFO:tasks.workunit.client.0.smithi060.stdout: "target_version": "0'0" 2020-08-24T01:03:59.892 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.893 INFO:tasks.workunit.client.0.smithi060.stdout: "last_change": "28", 2020-08-24T01:03:59.893 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend": "0", 2020-08-24T01:03:59.893 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_prenautilus": "0", 2020-08-24T01:03:59.893 INFO:tasks.workunit.client.0.smithi060.stdout: "last_force_op_resend_preluminous": "0", 2020-08-24T01:03:59.893 INFO:tasks.workunit.client.0.smithi060.stdout: "auid": 0, 2020-08-24T01:03:59.894 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_mode": "pool", 2020-08-24T01:03:59.894 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_seq": 1, 2020-08-24T01:03:59.894 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_epoch": 28, 2020-08-24T01:03:59.894 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_snaps": [ 2020-08-24T01:03:59.894 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.895 INFO:tasks.workunit.client.0.smithi060.stdout: "snapid": 1, 2020-08-24T01:03:59.895 INFO:tasks.workunit.client.0.smithi060.stdout: "stamp": "2020-08-24T01:02:18.063143+0000", 2020-08-24T01:03:59.895 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "snap1" 2020-08-24T01:03:59.895 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.895 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.895 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps": "[]", 2020-08-24T01:03:59.896 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_bytes": 0, 2020-08-24T01:03:59.896 INFO:tasks.workunit.client.0.smithi060.stdout: "quota_max_objects": 0, 2020-08-24T01:03:59.896 INFO:tasks.workunit.client.0.smithi060.stdout: "tiers": [], 2020-08-24T01:03:59.896 INFO:tasks.workunit.client.0.smithi060.stdout: "tier_of": -1, 2020-08-24T01:03:59.896 INFO:tasks.workunit.client.0.smithi060.stdout: "read_tier": -1, 2020-08-24T01:03:59.897 INFO:tasks.workunit.client.0.smithi060.stdout: "write_tier": -1, 2020-08-24T01:03:59.897 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_mode": "none", 2020-08-24T01:03:59.897 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_bytes": 0, 2020-08-24T01:03:59.897 INFO:tasks.workunit.client.0.smithi060.stdout: "target_max_objects": 0, 2020-08-24T01:03:59.897 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-08-24T01:03:59.898 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-08-24T01:03:59.898 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_target_full_ratio_micro": 800000, 2020-08-24T01:03:59.898 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_flush_age": 0, 2020-08-24T01:03:59.898 INFO:tasks.workunit.client.0.smithi060.stdout: "cache_min_evict_age": 0, 2020-08-24T01:03:59.898 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profile": "myprofile", 2020-08-24T01:03:59.898 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_params": { 2020-08-24T01:03:59.899 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "none" 2020-08-24T01:03:59.899 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.899 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_period": 0, 2020-08-24T01:03:59.899 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_count": 0, 2020-08-24T01:03:59.899 INFO:tasks.workunit.client.0.smithi060.stdout: "use_gmt_hitset": true, 2020-08-24T01:03:59.900 INFO:tasks.workunit.client.0.smithi060.stdout: "min_read_recency_for_promote": 0, 2020-08-24T01:03:59.900 INFO:tasks.workunit.client.0.smithi060.stdout: "min_write_recency_for_promote": 0, 2020-08-24T01:03:59.900 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_grade_decay_rate": 0, 2020-08-24T01:03:59.900 INFO:tasks.workunit.client.0.smithi060.stdout: "hit_set_search_last_n": 0, 2020-08-24T01:03:59.900 INFO:tasks.workunit.client.0.smithi060.stdout: "grade_table": [], 2020-08-24T01:03:59.900 INFO:tasks.workunit.client.0.smithi060.stdout: "stripe_width": 8192, 2020-08-24T01:03:59.901 INFO:tasks.workunit.client.0.smithi060.stdout: "expected_num_objects": 0, 2020-08-24T01:03:59.901 INFO:tasks.workunit.client.0.smithi060.stdout: "fast_read": false, 2020-08-24T01:03:59.901 INFO:tasks.workunit.client.0.smithi060.stdout: "options": {}, 2020-08-24T01:03:59.901 INFO:tasks.workunit.client.0.smithi060.stdout: "application_metadata": {} 2020-08-24T01:03:59.901 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.902 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.902 INFO:tasks.workunit.client.0.smithi060.stdout: "osds": [ 2020-08-24T01:03:59.902 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.902 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:03:59.902 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "2590757d-b008-4deb-b066-fccba9726114", 2020-08-24T01:03:59.903 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T01:03:59.903 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:03:59.903 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:03:59.903 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:03:59.903 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 34, 2020-08-24T01:03:59.903 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 38, 2020-08-24T01:03:59.904 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 40, 2020-08-24T01:03:59.904 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 43, 2020-08-24T01:03:59.904 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 39, 2020-08-24T01:03:59.904 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:03:59.904 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:59.905 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.905 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.905 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.905 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6802", 2020-08-24T01:03:59.905 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.905 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.906 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.906 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.906 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6803", 2020-08-24T01:03:59.906 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.906 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.907 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.907 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.907 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:03:59.907 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.907 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.908 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.908 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6804", 2020-08-24T01:03:59.908 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.908 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.908 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.908 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.909 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6805", 2020-08-24T01:03:59.909 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.909 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.909 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.909 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.910 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:03:59.910 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.910 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.910 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.910 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6808", 2020-08-24T01:03:59.911 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.911 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.911 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.911 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.911 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6809", 2020-08-24T01:03:59.911 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.912 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.912 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.912 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.912 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:03:59.912 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.913 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.913 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.913 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6806", 2020-08-24T01:03:59.913 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.913 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.913 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.914 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.914 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6807", 2020-08-24T01:03:59.914 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44430 2020-08-24T01:03:59.914 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.914 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.915 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.915 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6803/44430", 2020-08-24T01:03:59.915 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6805/44430", 2020-08-24T01:03:59.915 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6809/44430", 2020-08-24T01:03:59.915 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6807/44430", 2020-08-24T01:03:59.916 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:03:59.916 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T01:03:59.916 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T01:03:59.916 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.916 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.916 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.917 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:03:59.917 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "eb457a2e-e0bb-468b-864e-4386438ddbc4", 2020-08-24T01:03:59.917 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 0, 2020-08-24T01:03:59.917 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:03:59.917 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:03:59.918 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:03:59.918 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 0, 2020-08-24T01:03:59.918 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 0, 2020-08-24T01:03:59.918 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 9, 2020-08-24T01:03:59.918 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 23, 2020-08-24T01:03:59.919 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 30, 2020-08-24T01:03:59.919 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:03:59.919 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:59.919 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.919 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.919 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.920 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T01:03:59.920 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.920 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.920 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.920 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.921 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T01:03:59.921 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.921 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.921 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.921 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.921 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:03:59.922 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.922 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.922 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.922 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T01:03:59.922 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.923 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.923 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.923 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.923 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T01:03:59.923 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.923 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.924 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.924 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.924 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:03:59.924 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.924 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.925 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.925 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T01:03:59.925 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.925 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.925 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.926 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.926 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T01:03:59.926 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.926 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.926 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.926 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.927 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:03:59.927 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.927 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.927 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.927 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T01:03:59.928 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.928 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.928 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.928 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.928 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T01:03:59.928 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 38050 2020-08-24T01:03:59.929 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.929 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.929 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.929 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/38050", 2020-08-24T01:03:59.929 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/38050", 2020-08-24T01:03:59.930 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/38050", 2020-08-24T01:03:59.930 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/38050", 2020-08-24T01:03:59.930 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:03:59.930 INFO:tasks.workunit.client.0.smithi060.stdout: "exists" 2020-08-24T01:03:59.930 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.931 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.931 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.931 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:03:59.931 INFO:tasks.workunit.client.0.smithi060.stdout: "uuid": "813ed60f-f06b-4b40-a01a-9681d9e7dd5d", 2020-08-24T01:03:59.931 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 1, 2020-08-24T01:03:59.931 INFO:tasks.workunit.client.0.smithi060.stdout: "in": 1, 2020-08-24T01:03:59.932 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 1, 2020-08-24T01:03:59.932 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_affinity": 1, 2020-08-24T01:03:59.932 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_begin": 35, 2020-08-24T01:03:59.932 INFO:tasks.workunit.client.0.smithi060.stdout: "last_clean_end": 38, 2020-08-24T01:03:59.932 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 43, 2020-08-24T01:03:59.933 INFO:tasks.workunit.client.0.smithi060.stdout: "up_thru": 0, 2020-08-24T01:03:59.933 INFO:tasks.workunit.client.0.smithi060.stdout: "down_at": 42, 2020-08-24T01:03:59.933 INFO:tasks.workunit.client.0.smithi060.stdout: "lost_at": 0, 2020-08-24T01:03:59.933 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addrs": { 2020-08-24T01:03:59.933 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.933 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.934 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.934 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6810", 2020-08-24T01:03:59.934 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.934 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.934 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.935 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.935 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6811", 2020-08-24T01:03:59.935 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.935 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.935 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.935 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.936 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addrs": { 2020-08-24T01:03:59.936 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.936 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.936 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.936 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6812", 2020-08-24T01:03:59.937 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.937 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.937 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.937 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.937 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6813", 2020-08-24T01:03:59.938 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.938 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.938 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.938 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.938 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addrs": { 2020-08-24T01:03:59.938 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.939 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.939 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.939 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6816", 2020-08-24T01:03:59.939 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.939 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.940 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.940 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.940 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6817", 2020-08-24T01:03:59.940 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.940 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.940 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.941 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.941 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addrs": { 2020-08-24T01:03:59.941 INFO:tasks.workunit.client.0.smithi060.stdout: "addrvec": [ 2020-08-24T01:03:59.941 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.941 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v2", 2020-08-24T01:03:59.942 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6814", 2020-08-24T01:03:59.942 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.942 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.942 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.942 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "v1", 2020-08-24T01:03:59.942 INFO:tasks.workunit.client.0.smithi060.stdout: "addr": "127.0.0.1:6815", 2020-08-24T01:03:59.943 INFO:tasks.workunit.client.0.smithi060.stdout: "nonce": 44662 2020-08-24T01:03:59.943 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.943 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.943 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.943 INFO:tasks.workunit.client.0.smithi060.stdout: "public_addr": "127.0.0.1:6811/44662", 2020-08-24T01:03:59.944 INFO:tasks.workunit.client.0.smithi060.stdout: "cluster_addr": "127.0.0.1:6813/44662", 2020-08-24T01:03:59.944 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_back_addr": "127.0.0.1:6817/44662", 2020-08-24T01:03:59.944 INFO:tasks.workunit.client.0.smithi060.stdout: "heartbeat_front_addr": "127.0.0.1:6815/44662", 2020-08-24T01:03:59.944 INFO:tasks.workunit.client.0.smithi060.stdout: "state": [ 2020-08-24T01:03:59.944 INFO:tasks.workunit.client.0.smithi060.stdout: "exists", 2020-08-24T01:03:59.945 INFO:tasks.workunit.client.0.smithi060.stdout: "up" 2020-08-24T01:03:59.945 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.945 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.945 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.945 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_xinfo": [ 2020-08-24T01:03:59.945 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.946 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:03:59.946 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T01:03:22.243212+0000", 2020-08-24T01:03:59.946 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:03:59.946 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:03:59.946 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:03:59.947 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:03:59.947 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T01:01:20.164740+0000", 2020-08-24T01:03:59.947 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:03:59.947 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.947 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.947 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:03:59.948 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T01:02:29.559713+0000", 2020-08-24T01:03:59.948 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:03:59.948 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:03:59.948 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:03:59.948 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:03:59.949 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T01:01:26.672227+0000", 2020-08-24T01:03:59.949 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:03:59.949 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.949 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.949 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:03:59.950 INFO:tasks.workunit.client.0.smithi060.stdout: "down_stamp": "2020-08-24T01:03:24.386047+0000", 2020-08-24T01:03:59.950 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_probability": 0, 2020-08-24T01:03:59.950 INFO:tasks.workunit.client.0.smithi060.stdout: "laggy_interval": 0, 2020-08-24T01:03:59.950 INFO:tasks.workunit.client.0.smithi060.stdout: "features": 4540138292836696063, 2020-08-24T01:03:59.950 INFO:tasks.workunit.client.0.smithi060.stdout: "old_weight": 0, 2020-08-24T01:03:59.950 INFO:tasks.workunit.client.0.smithi060.stdout: "last_purged_snaps_scrub": "2020-08-24T01:01:33.366933+0000", 2020-08-24T01:03:59.951 INFO:tasks.workunit.client.0.smithi060.stdout: "dead_epoch": 0 2020-08-24T01:03:59.951 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.951 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.951 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap": [], 2020-08-24T01:03:59.951 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_upmap_items": [], 2020-08-24T01:03:59.952 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_temp": [], 2020-08-24T01:03:59.952 INFO:tasks.workunit.client.0.smithi060.stdout: "primary_temp": [], 2020-08-24T01:03:59.952 INFO:tasks.workunit.client.0.smithi060.stdout: "blacklist": {}, 2020-08-24T01:03:59.952 INFO:tasks.workunit.client.0.smithi060.stdout: "erasure_code_profiles": { 2020-08-24T01:03:59.952 INFO:tasks.workunit.client.0.smithi060.stdout: "default": { 2020-08-24T01:03:59.953 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T01:03:59.953 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "2", 2020-08-24T01:03:59.953 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T01:03:59.953 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van" 2020-08-24T01:03:59.953 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.953 INFO:tasks.workunit.client.0.smithi060.stdout: "myprofile": { 2020-08-24T01:03:59.954 INFO:tasks.workunit.client.0.smithi060.stdout: "crush-device-class": "", 2020-08-24T01:03:59.954 INFO:tasks.workunit.client.0.smithi060.stdout: "crush-failure-domain": "osd", 2020-08-24T01:03:59.954 INFO:tasks.workunit.client.0.smithi060.stdout: "crush-root": "default", 2020-08-24T01:03:59.954 INFO:tasks.workunit.client.0.smithi060.stdout: "jerasure-per-chunk-alignment": "false", 2020-08-24T01:03:59.954 INFO:tasks.workunit.client.0.smithi060.stdout: "k": "2", 2020-08-24T01:03:59.955 INFO:tasks.workunit.client.0.smithi060.stdout: "m": "1", 2020-08-24T01:03:59.955 INFO:tasks.workunit.client.0.smithi060.stdout: "plugin": "jerasure", 2020-08-24T01:03:59.955 INFO:tasks.workunit.client.0.smithi060.stdout: "technique": "reed_sol_van", 2020-08-24T01:03:59.955 INFO:tasks.workunit.client.0.smithi060.stdout: "w": "8" 2020-08-24T01:03:59.955 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.955 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.956 INFO:tasks.workunit.client.0.smithi060.stdout: "removed_snaps_queue": [], 2020-08-24T01:03:59.956 INFO:tasks.workunit.client.0.smithi060.stdout: "new_removed_snaps": [], 2020-08-24T01:03:59.956 INFO:tasks.workunit.client.0.smithi060.stdout: "new_purged_snaps": [], 2020-08-24T01:03:59.956 INFO:tasks.workunit.client.0.smithi060.stdout: "crush_node_flags": {}, 2020-08-24T01:03:59.956 INFO:tasks.workunit.client.0.smithi060.stdout: "device_class_flags": {} 2020-08-24T01:03:59.957 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.957 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_metadata": [ 2020-08-24T01:03:59.957 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.957 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:03:59.957 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:03:59.958 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6804/44430,v1:127.0.0.1:6805/44430]", 2020-08-24T01:03:59.958 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:03:59.958 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:03:59.958 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:03:59.958 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:03:59.958 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:03:59.959 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:03:59.959 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:03:59.959 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:03:59.959 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:03:59.959 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-08-24T01:03:59.960 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:03:59.960 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:03:59.960 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:03:59.960 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:03:59.960 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:03:59.960 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:03:59.961 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:03:59.961 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:03:59.961 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:03:59.962 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:03:59.962 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:03:59.962 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:03:59.962 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:03:59.963 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:03:59.963 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:03:59.963 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6802/44430,v1:127.0.0.1:6803/44430]", 2020-08-24T01:03:59.963 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:03:59.963 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/44430,v1:127.0.0.1:6809/44430]", 2020-08-24T01:03:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/44430,v1:127.0.0.1:6807/44430]", 2020-08-24T01:03:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:03:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:03:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:03:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:03:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:03:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:03:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:03:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:03:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:03:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/0", 2020-08-24T01:03:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:03:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:03:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:03:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:03:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:03:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/38050,v1:127.0.0.1:6813/38050]", 2020-08-24T01:03:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:03:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:03:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:03:59.968 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:03:59.968 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:03:59.968 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:03:59.968 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:03:59.968 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:03:59.969 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:03:59.969 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-08-24T01:03:59.969 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:03:59.969 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:03:59.969 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:03:59.970 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:03:59.970 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:03:59.970 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:03:59.970 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:03:59.970 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:03:59.970 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:03:59.971 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:03:59.971 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:03:59.971 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:03:59.971 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:03:59.971 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:03:59.972 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:03:59.972 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/38050,v1:127.0.0.1:6811/38050]", 2020-08-24T01:03:59.972 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:03:59.972 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/38050,v1:127.0.0.1:6817/38050]", 2020-08-24T01:03:59.972 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/38050,v1:127.0.0.1:6815/38050]", 2020-08-24T01:03:59.972 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:03:59.973 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:03:59.973 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:03:59.973 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:03:59.973 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:03:59.973 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:03:59.974 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:03:59.974 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:03:59.974 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:03:59.974 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/1", 2020-08-24T01:03:59.974 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:03:59.974 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:03:59.975 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:03:59.975 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.975 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.976 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:03:59.977 INFO:tasks.workunit.client.0.smithi060.stdout: "arch": "x86_64", 2020-08-24T01:03:59.977 INFO:tasks.workunit.client.0.smithi060.stdout: "back_addr": "[v2:127.0.0.1:6812/44662,v1:127.0.0.1:6813/44662]", 2020-08-24T01:03:59.977 INFO:tasks.workunit.client.0.smithi060.stdout: "back_iface": "", 2020-08-24T01:03:59.978 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs": "1", 2020-08-24T01:03:59.978 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_db": "0", 2020-08-24T01:03:59.978 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_dedicated_wal": "0", 2020-08-24T01:03:59.979 INFO:tasks.workunit.client.0.smithi060.stdout: "bluefs_single_shared_device": "1", 2020-08-24T01:03:59.979 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_access_mode": "file", 2020-08-24T01:03:59.979 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_block_size": "4096", 2020-08-24T01:03:59.980 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_devices": "sda", 2020-08-24T01:03:59.980 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-08-24T01:03:59.980 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-08-24T01:03:59.981 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_rotational": "1", 2020-08-24T01:03:59.981 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_size": "107374182400", 2020-08-24T01:03:59.981 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_support_discard": "0", 2020-08-24T01:03:59.982 INFO:tasks.workunit.client.0.smithi060.stdout: "bluestore_bdev_type": "hdd", 2020-08-24T01:03:59.982 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_release": "octopus", 2020-08-24T01:03:59.982 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version": "ceph version 15.2.4-661-g0a941e5 (0a941e55043ccee6d35a8717bec55ff7581e67d8) octopus (stable)", 2020-08-24T01:03:59.983 INFO:tasks.workunit.client.0.smithi060.stdout: "ceph_version_short": "15.2.4-661-g0a941e5", 2020-08-24T01:03:59.983 INFO:tasks.workunit.client.0.smithi060.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz", 2020-08-24T01:03:59.983 INFO:tasks.workunit.client.0.smithi060.stdout: "default_device_class": "hdd", 2020-08-24T01:03:59.984 INFO:tasks.workunit.client.0.smithi060.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W4H4ED", 2020-08-24T01:03:59.984 INFO:tasks.workunit.client.0.smithi060.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-08-24T01:03:59.984 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": "sda", 2020-08-24T01:03:59.985 INFO:tasks.workunit.client.0.smithi060.stdout: "distro": "ubuntu", 2020-08-24T01:03:59.985 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_description": "Ubuntu 18.04.5 LTS", 2020-08-24T01:03:59.985 INFO:tasks.workunit.client.0.smithi060.stdout: "distro_version": "18.04", 2020-08-24T01:03:59.986 INFO:tasks.workunit.client.0.smithi060.stdout: "front_addr": "[v2:127.0.0.1:6810/44662,v1:127.0.0.1:6811/44662]", 2020-08-24T01:03:59.986 INFO:tasks.workunit.client.0.smithi060.stdout: "front_iface": "", 2020-08-24T01:03:59.987 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/44662,v1:127.0.0.1:6817/44662]", 2020-08-24T01:03:59.987 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/44662,v1:127.0.0.1:6815/44662]", 2020-08-24T01:03:59.987 INFO:tasks.workunit.client.0.smithi060.stdout: "hostname": "smithi060", 2020-08-24T01:03:59.988 INFO:tasks.workunit.client.0.smithi060.stdout: "journal_rotational": "1", 2020-08-24T01:03:59.988 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_description": "#113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020", 2020-08-24T01:03:59.988 INFO:tasks.workunit.client.0.smithi060.stdout: "kernel_version": "4.15.0-112-generic", 2020-08-24T01:03:59.989 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_swap_kb": "2097148", 2020-08-24T01:03:59.989 INFO:tasks.workunit.client.0.smithi060.stdout: "mem_total_kb": "32830264", 2020-08-24T01:03:59.989 INFO:tasks.workunit.client.0.smithi060.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-08-24T01:03:59.989 INFO:tasks.workunit.client.0.smithi060.stdout: "objectstore_numa_unknown_devices": "sda", 2020-08-24T01:03:59.990 INFO:tasks.workunit.client.0.smithi060.stdout: "os": "Linux", 2020-08-24T01:03:59.990 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_data": "td/divergent-priors/2", 2020-08-24T01:03:59.990 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_objectstore": "bluestore", 2020-08-24T01:03:59.991 INFO:tasks.workunit.client.0.smithi060.stdout: "osdspec_affinity": "", 2020-08-24T01:03:59.991 INFO:tasks.workunit.client.0.smithi060.stdout: "rotational": "1" 2020-08-24T01:03:59.991 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.992 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:03:59.992 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_clean_epochs": { 2020-08-24T01:03:59.992 INFO:tasks.workunit.client.0.smithi060.stdout: "min_last_epoch_clean": 0, 2020-08-24T01:03:59.992 INFO:tasks.workunit.client.0.smithi060.stdout: "last_epoch_clean": { 2020-08-24T01:03:59.993 INFO:tasks.workunit.client.0.smithi060.stdout: "per_pool": [] 2020-08-24T01:03:59.993 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.993 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_epochs": [ 2020-08-24T01:03:59.994 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.994 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:03:59.994 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 41 2020-08-24T01:03:59.995 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.995 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.995 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:03:59.995 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 9 2020-08-24T01:03:59.996 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.996 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:03:59.997 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:03:59.997 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 44 2020-08-24T01:03:59.997 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:03:59.998 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:03:59.998 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:03:59.998 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_first_committed": 1, 2020-08-24T01:03:59.998 INFO:tasks.workunit.client.0.smithi060.stdout: "osdmap_last_committed": 44, 2020-08-24T01:03:59.999 INFO:tasks.workunit.client.0.smithi060.stdout: "crushmap": { 2020-08-24T01:03:59.999 INFO:tasks.workunit.client.0.smithi060.stdout: "devices": [ 2020-08-24T01:03:59.999 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.000 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:04:00.000 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.0", 2020-08-24T01:04:00.000 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:04:00.001 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.001 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.001 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:04:00.002 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.1", 2020-08-24T01:04:00.002 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:04:00.002 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.002 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.003 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:04:00.003 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd.2", 2020-08-24T01:04:00.003 INFO:tasks.workunit.client.0.smithi060.stdout: "class": "hdd" 2020-08-24T01:04:00.004 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.004 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.004 INFO:tasks.workunit.client.0.smithi060.stdout: "types": [ 2020-08-24T01:04:00.004 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.005 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 0, 2020-08-24T01:04:00.005 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "osd" 2020-08-24T01:04:00.005 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.006 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.006 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:04:00.006 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "host" 2020-08-24T01:04:00.006 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.007 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.007 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 2, 2020-08-24T01:04:00.007 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "chassis" 2020-08-24T01:04:00.008 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.008 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.008 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 3, 2020-08-24T01:04:00.008 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "rack" 2020-08-24T01:04:00.009 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.009 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.009 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 4, 2020-08-24T01:04:00.010 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "row" 2020-08-24T01:04:00.010 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.010 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.011 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 5, 2020-08-24T01:04:00.011 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pdu" 2020-08-24T01:04:00.011 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.011 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.012 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 6, 2020-08-24T01:04:00.012 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "pod" 2020-08-24T01:04:00.012 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.013 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.013 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 7, 2020-08-24T01:04:00.013 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "room" 2020-08-24T01:04:00.013 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.014 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.014 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 8, 2020-08-24T01:04:00.015 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "datacenter" 2020-08-24T01:04:00.015 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.015 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.016 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 9, 2020-08-24T01:04:00.016 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "zone" 2020-08-24T01:04:00.016 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.017 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.017 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 10, 2020-08-24T01:04:00.017 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "region" 2020-08-24T01:04:00.018 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.018 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.018 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:04:00.018 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "root" 2020-08-24T01:04:00.019 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.019 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.019 INFO:tasks.workunit.client.0.smithi060.stdout: "buckets": [ 2020-08-24T01:04:00.019 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.019 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -1, 2020-08-24T01:04:00.020 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default", 2020-08-24T01:04:00.020 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:04:00.020 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T01:04:00.020 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:04:00.021 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:04:00.021 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:04:00.021 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:04:00.021 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.022 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T01:04:00.022 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:04:00.022 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:04:00.022 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.023 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:04:00.023 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.023 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.023 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -2, 2020-08-24T01:04:00.024 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "default~hdd", 2020-08-24T01:04:00.024 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 11, 2020-08-24T01:04:00.024 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "root", 2020-08-24T01:04:00.025 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:04:00.025 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:04:00.025 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:04:00.026 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:04:00.026 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.026 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T01:04:00.027 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:04:00.027 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:04:00.027 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.028 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:04:00.028 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.028 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.028 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -3, 2020-08-24T01:04:00.028 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060", 2020-08-24T01:04:00.029 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:04:00.029 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T01:04:00.029 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:04:00.029 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:04:00.029 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:04:00.029 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:04:00.030 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.030 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:04:00.030 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:04:00.031 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:04:00.031 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.031 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.031 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:04:00.031 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:04:00.032 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T01:04:00.032 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.032 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.032 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:04:00.032 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:04:00.033 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T01:04:00.033 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.033 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:04:00.033 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.033 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.033 INFO:tasks.workunit.client.0.smithi060.stdout: "id": -4, 2020-08-24T01:04:00.034 INFO:tasks.workunit.client.0.smithi060.stdout: "name": "smithi060~hdd", 2020-08-24T01:04:00.034 INFO:tasks.workunit.client.0.smithi060.stdout: "type_id": 1, 2020-08-24T01:04:00.034 INFO:tasks.workunit.client.0.smithi060.stdout: "type_name": "host", 2020-08-24T01:04:00.034 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 19206, 2020-08-24T01:04:00.034 INFO:tasks.workunit.client.0.smithi060.stdout: "alg": "straw2", 2020-08-24T01:04:00.035 INFO:tasks.workunit.client.0.smithi060.stdout: "hash": "rjenkins1", 2020-08-24T01:04:00.035 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:04:00.035 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.035 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 0, 2020-08-24T01:04:00.035 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:04:00.036 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 0 2020-08-24T01:04:00.036 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.036 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.036 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 1, 2020-08-24T01:04:00.036 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:04:00.036 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 1 2020-08-24T01:04:00.037 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.037 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.037 INFO:tasks.workunit.client.0.smithi060.stdout: "id": 2, 2020-08-24T01:04:00.037 INFO:tasks.workunit.client.0.smithi060.stdout: "weight": 6402, 2020-08-24T01:04:00.037 INFO:tasks.workunit.client.0.smithi060.stdout: "pos": 2 2020-08-24T01:04:00.038 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.038 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:04:00.038 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.039 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.039 INFO:tasks.workunit.client.0.smithi060.stdout: "rules": [ 2020-08-24T01:04:00.039 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.039 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 0, 2020-08-24T01:04:00.039 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "replicated_rule", 2020-08-24T01:04:00.040 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 0, 2020-08-24T01:04:00.040 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 1, 2020-08-24T01:04:00.040 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 1, 2020-08-24T01:04:00.040 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 10, 2020-08-24T01:04:00.040 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T01:04:00.040 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.041 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T01:04:00.041 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T01:04:00.041 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T01:04:00.041 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.041 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.042 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_firstn", 2020-08-24T01:04:00.042 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T01:04:00.042 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T01:04:00.042 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.042 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.043 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T01:04:00.043 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.043 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:04:00.043 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.043 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.043 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_id": 1, 2020-08-24T01:04:00.044 INFO:tasks.workunit.client.0.smithi060.stdout: "rule_name": "test", 2020-08-24T01:04:00.044 INFO:tasks.workunit.client.0.smithi060.stdout: "ruleset": 1, 2020-08-24T01:04:00.044 INFO:tasks.workunit.client.0.smithi060.stdout: "type": 3, 2020-08-24T01:04:00.044 INFO:tasks.workunit.client.0.smithi060.stdout: "min_size": 3, 2020-08-24T01:04:00.044 INFO:tasks.workunit.client.0.smithi060.stdout: "max_size": 3, 2020-08-24T01:04:00.045 INFO:tasks.workunit.client.0.smithi060.stdout: "steps": [ 2020-08-24T01:04:00.045 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.045 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "set_chooseleaf_tries", 2020-08-24T01:04:00.045 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 5 2020-08-24T01:04:00.045 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.046 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.046 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "set_choose_tries", 2020-08-24T01:04:00.046 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 100 2020-08-24T01:04:00.046 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.046 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.047 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "take", 2020-08-24T01:04:00.047 INFO:tasks.workunit.client.0.smithi060.stdout: "item": -1, 2020-08-24T01:04:00.047 INFO:tasks.workunit.client.0.smithi060.stdout: "item_name": "default" 2020-08-24T01:04:00.047 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.048 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.048 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "choose_indep", 2020-08-24T01:04:00.048 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 0, 2020-08-24T01:04:00.049 INFO:tasks.workunit.client.0.smithi060.stdout: "type": "osd" 2020-08-24T01:04:00.049 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.049 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.050 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "emit" 2020-08-24T01:04:00.050 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.050 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:04:00.050 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.051 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.051 INFO:tasks.workunit.client.0.smithi060.stdout: "tunables": { 2020-08-24T01:04:00.051 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_tries": 0, 2020-08-24T01:04:00.051 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_local_fallback_tries": 0, 2020-08-24T01:04:00.051 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_total_tries": 50, 2020-08-24T01:04:00.052 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_descend_once": 1, 2020-08-24T01:04:00.052 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_vary_r": 1, 2020-08-24T01:04:00.052 INFO:tasks.workunit.client.0.smithi060.stdout: "chooseleaf_stable": 1, 2020-08-24T01:04:00.052 INFO:tasks.workunit.client.0.smithi060.stdout: "straw_calc_version": 1, 2020-08-24T01:04:00.052 INFO:tasks.workunit.client.0.smithi060.stdout: "allowed_bucket_algs": 54, 2020-08-24T01:04:00.053 INFO:tasks.workunit.client.0.smithi060.stdout: "profile": "jewel", 2020-08-24T01:04:00.053 INFO:tasks.workunit.client.0.smithi060.stdout: "optimal_tunables": 1, 2020-08-24T01:04:00.053 INFO:tasks.workunit.client.0.smithi060.stdout: "legacy_tunables": 0, 2020-08-24T01:04:00.053 INFO:tasks.workunit.client.0.smithi060.stdout: "minimum_required_version": "jewel", 2020-08-24T01:04:00.053 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables": 1, 2020-08-24T01:04:00.054 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables2": 1, 2020-08-24T01:04:00.054 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v2_rules": 1, 2020-08-24T01:04:00.054 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables3": 1, 2020-08-24T01:04:00.054 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v3_rules": 0, 2020-08-24T01:04:00.054 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v4_buckets": 1, 2020-08-24T01:04:00.055 INFO:tasks.workunit.client.0.smithi060.stdout: "require_feature_tunables5": 1, 2020-08-24T01:04:00.055 INFO:tasks.workunit.client.0.smithi060.stdout: "has_v5_rules": 0 2020-08-24T01:04:00.055 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.055 INFO:tasks.workunit.client.0.smithi060.stdout: "choose_args": {} 2020-08-24T01:04:00.056 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.056 INFO:tasks.workunit.client.0.smithi060.stdout: "fsmap": { 2020-08-24T01:04:00.056 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:04:00.056 INFO:tasks.workunit.client.0.smithi060.stdout: "default_fscid": -1, 2020-08-24T01:04:00.056 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": { 2020-08-24T01:04:00.057 INFO:tasks.workunit.client.0.smithi060.stdout: "compat": {}, 2020-08-24T01:04:00.057 INFO:tasks.workunit.client.0.smithi060.stdout: "ro_compat": {}, 2020-08-24T01:04:00.057 INFO:tasks.workunit.client.0.smithi060.stdout: "incompat": { 2020-08-24T01:04:00.057 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_1": "base v0.20", 2020-08-24T01:04:00.057 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_2": "client writeable ranges", 2020-08-24T01:04:00.058 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_3": "default file layouts on dirs", 2020-08-24T01:04:00.058 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_4": "dir inode in separate object", 2020-08-24T01:04:00.058 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_5": "mds uses versioned encoding", 2020-08-24T01:04:00.058 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_6": "dirfrag is stored in omap", 2020-08-24T01:04:00.058 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_8": "no anchor table", 2020-08-24T01:04:00.059 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_9": "file layout v2", 2020-08-24T01:04:00.059 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_10": "snaprealm v2" 2020-08-24T01:04:00.059 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.059 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.060 INFO:tasks.workunit.client.0.smithi060.stdout: "feature_flags": { 2020-08-24T01:04:00.060 INFO:tasks.workunit.client.0.smithi060.stdout: "enable_multiple": false, 2020-08-24T01:04:00.061 INFO:tasks.workunit.client.0.smithi060.stdout: "ever_enabled_multiple": false 2020-08-24T01:04:00.061 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.062 INFO:tasks.workunit.client.0.smithi060.stdout: "standbys": [], 2020-08-24T01:04:00.062 INFO:tasks.workunit.client.0.smithi060.stdout: "filesystems": [] 2020-08-24T01:04:00.062 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.063 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_first_committed": 1, 2020-08-24T01:04:00.063 INFO:tasks.workunit.client.0.smithi060.stdout: "mdsmap_last_committed": 1, 2020-08-24T01:04:00.063 INFO:tasks.workunit.client.0.smithi060.stdout: "auth": { 2020-08-24T01:04:00.064 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T01:04:00.064 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 5, 2020-08-24T01:04:00.064 INFO:tasks.workunit.client.0.smithi060.stdout: "num_secrets": 3 2020-08-24T01:04:00.065 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.065 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T01:04:00.065 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_active": 1, 2020-08-24T01:04:00.066 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_unknown": 0, 2020-08-24T01:04:00.066 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osd": 3, 2020-08-24T01:04:00.066 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_sum": { 2020-08-24T01:04:00.067 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T01:04:00.067 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 62471, 2020-08-24T01:04:00.067 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 101, 2020-08-24T01:04:00.067 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 1, 2020-08-24T01:04:00.068 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 303, 2020-08-24T01:04:00.068 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T01:04:00.068 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T01:04:00.069 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 101, 2020-08-24T01:04:00.069 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T01:04:00.069 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T01:04:00.070 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 101, 2020-08-24T01:04:00.070 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T01:04:00.070 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T01:04:00.071 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T01:04:00.072 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T01:04:00.072 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T01:04:00.072 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T01:04:00.073 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T01:04:00.073 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T01:04:00.074 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T01:04:00.074 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T01:04:00.075 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T01:04:00.075 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T01:04:00.075 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T01:04:00.076 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T01:04:00.076 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T01:04:00.076 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T01:04:00.077 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T01:04:00.077 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T01:04:00.077 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T01:04:00.078 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T01:04:00.078 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T01:04:00.078 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T01:04:00.078 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T01:04:00.078 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T01:04:00.079 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T01:04:00.079 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T01:04:00.079 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T01:04:00.079 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T01:04:00.079 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T01:04:00.080 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T01:04:00.080 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.080 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T01:04:00.080 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T01:04:00.080 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T01:04:00.081 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T01:04:00.081 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 0, 2020-08-24T01:04:00.081 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 0, 2020-08-24T01:04:00.082 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:04:00.082 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:04:00.082 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:04:00.082 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T01:04:00.082 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T01:04:00.083 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.083 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 18, 2020-08-24T01:04:00.083 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 18, 2020-08-24T01:04:00.083 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 3, 2020-08-24T01:04:00.083 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 3, 2020-08-24T01:04:00.084 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 0 2020-08-24T01:04:00.084 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.084 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum": { 2020-08-24T01:04:00.085 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T01:04:00.085 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T01:04:00.085 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T01:04:00.086 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T01:04:00.086 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T01:04:00.086 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T01:04:00.087 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T01:04:00.087 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3180672, 2020-08-24T01:04:00.087 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 34624, 2020-08-24T01:04:00.088 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 8, 2020-08-24T01:04:00.088 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145719, 2020-08-24T01:04:00.088 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311392128, 2020-08-24T01:04:00.089 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T01:04:00.089 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T01:04:00.089 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318865539072, 2020-08-24T01:04:00.090 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T01:04:00.090 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 35454976, 2020-08-24T01:04:00.090 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 1550633, 2020-08-24T01:04:00.091 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:04:00.091 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:04:00.091 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:04:00.091 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 8536, 2020-08-24T01:04:00.092 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221216936 2020-08-24T01:04:00.092 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.093 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T01:04:00.093 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T01:04:00.093 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T01:04:00.093 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T01:04:00.094 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T01:04:00.094 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T01:04:00.094 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T01:04:00.094 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.095 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T01:04:00.095 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T01:04:00.095 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T01:04:00.095 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T01:04:00.096 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T01:04:00.096 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.096 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T01:04:00.096 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T01:04:00.097 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.097 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_sum_by_class": { 2020-08-24T01:04:00.097 INFO:tasks.workunit.client.0.smithi060.stdout: "hdd": { 2020-08-24T01:04:00.097 INFO:tasks.workunit.client.0.smithi060.stdout: "up_from": 0, 2020-08-24T01:04:00.098 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 0, 2020-08-24T01:04:00.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pgs": 3, 2020-08-24T01:04:00.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_osds": 3, 2020-08-24T01:04:00.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_osds": 3, 2020-08-24T01:04:00.098 INFO:tasks.workunit.client.0.smithi060.stdout: "num_per_pool_omap_osds": 3, 2020-08-24T01:04:00.099 INFO:tasks.workunit.client.0.smithi060.stdout: "kb": 314572800, 2020-08-24T01:04:00.099 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used": 3180672, 2020-08-24T01:04:00.099 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_data": 34624, 2020-08-24T01:04:00.099 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_omap": 8, 2020-08-24T01:04:00.099 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_used_meta": 3145719, 2020-08-24T01:04:00.099 INFO:tasks.workunit.client.0.smithi060.stdout: "kb_avail": 311392128, 2020-08-24T01:04:00.100 INFO:tasks.workunit.client.0.smithi060.stdout: "statfs": { 2020-08-24T01:04:00.100 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 322122547200, 2020-08-24T01:04:00.100 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 318865539072, 2020-08-24T01:04:00.100 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 3221225472, 2020-08-24T01:04:00.101 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 35454976, 2020-08-24T01:04:00.101 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 1550633, 2020-08-24T01:04:00.101 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:04:00.101 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:04:00.102 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:04:00.102 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 8536, 2020-08-24T01:04:00.103 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 3221216936 2020-08-24T01:04:00.103 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.103 INFO:tasks.workunit.client.0.smithi060.stdout: "hb_peers": [], 2020-08-24T01:04:00.103 INFO:tasks.workunit.client.0.smithi060.stdout: "snap_trim_queue_len": 0, 2020-08-24T01:04:00.104 INFO:tasks.workunit.client.0.smithi060.stdout: "num_snap_trimming": 0, 2020-08-24T01:04:00.104 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shards_repaired": 0, 2020-08-24T01:04:00.104 INFO:tasks.workunit.client.0.smithi060.stdout: "op_queue_age_hist": { 2020-08-24T01:04:00.104 INFO:tasks.workunit.client.0.smithi060.stdout: "histogram": [], 2020-08-24T01:04:00.104 INFO:tasks.workunit.client.0.smithi060.stdout: "upper_bound": 1 2020-08-24T01:04:00.105 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.105 INFO:tasks.workunit.client.0.smithi060.stdout: "perf_stat": { 2020-08-24T01:04:00.105 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ms": 0, 2020-08-24T01:04:00.105 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ms": 0, 2020-08-24T01:04:00.105 INFO:tasks.workunit.client.0.smithi060.stdout: "commit_latency_ns": 0, 2020-08-24T01:04:00.106 INFO:tasks.workunit.client.0.smithi060.stdout: "apply_latency_ns": 0 2020-08-24T01:04:00.106 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.106 INFO:tasks.workunit.client.0.smithi060.stdout: "alerts": [], 2020-08-24T01:04:00.106 INFO:tasks.workunit.client.0.smithi060.stdout: "network_ping_times": [] 2020-08-24T01:04:00.107 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.107 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.107 INFO:tasks.workunit.client.0.smithi060.stdout: "pool_stats": [ 2020-08-24T01:04:00.107 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.107 INFO:tasks.workunit.client.0.smithi060.stdout: "poolid": 1, 2020-08-24T01:04:00.107 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg": 1, 2020-08-24T01:04:00.108 INFO:tasks.workunit.client.0.smithi060.stdout: "stat_sum": { 2020-08-24T01:04:00.108 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes": 62471, 2020-08-24T01:04:00.108 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects": 101, 2020-08-24T01:04:00.109 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_clones": 1, 2020-08-24T01:04:00.109 INFO:tasks.workunit.client.0.smithi060.stdout: "num_object_copies": 303, 2020-08-24T01:04:00.109 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing_on_primary": 0, 2020-08-24T01:04:00.110 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_missing": 0, 2020-08-24T01:04:00.110 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_degraded": 101, 2020-08-24T01:04:00.110 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_misplaced": 0, 2020-08-24T01:04:00.111 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_unfound": 0, 2020-08-24T01:04:00.111 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_dirty": 101, 2020-08-24T01:04:00.111 INFO:tasks.workunit.client.0.smithi060.stdout: "num_whiteouts": 0, 2020-08-24T01:04:00.111 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read": 0, 2020-08-24T01:04:00.112 INFO:tasks.workunit.client.0.smithi060.stdout: "num_read_kb": 0, 2020-08-24T01:04:00.112 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write": 101, 2020-08-24T01:04:00.112 INFO:tasks.workunit.client.0.smithi060.stdout: "num_write_kb": 101, 2020-08-24T01:04:00.112 INFO:tasks.workunit.client.0.smithi060.stdout: "num_scrub_errors": 0, 2020-08-24T01:04:00.112 INFO:tasks.workunit.client.0.smithi060.stdout: "num_shallow_scrub_errors": 0, 2020-08-24T01:04:00.113 INFO:tasks.workunit.client.0.smithi060.stdout: "num_deep_scrub_errors": 0, 2020-08-24T01:04:00.113 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_recovered": 0, 2020-08-24T01:04:00.114 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_recovered": 0, 2020-08-24T01:04:00.114 INFO:tasks.workunit.client.0.smithi060.stdout: "num_keys_recovered": 0, 2020-08-24T01:04:00.114 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_omap": 0, 2020-08-24T01:04:00.114 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_hit_set_archive": 0, 2020-08-24T01:04:00.114 INFO:tasks.workunit.client.0.smithi060.stdout: "num_bytes_hit_set_archive": 0, 2020-08-24T01:04:00.115 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush": 0, 2020-08-24T01:04:00.115 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_kb": 0, 2020-08-24T01:04:00.115 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict": 0, 2020-08-24T01:04:00.115 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_kb": 0, 2020-08-24T01:04:00.115 INFO:tasks.workunit.client.0.smithi060.stdout: "num_promote": 0, 2020-08-24T01:04:00.115 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_high": 0, 2020-08-24T01:04:00.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_flush_mode_low": 0, 2020-08-24T01:04:00.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_some": 0, 2020-08-24T01:04:00.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_evict_mode_full": 0, 2020-08-24T01:04:00.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_pinned": 0, 2020-08-24T01:04:00.116 INFO:tasks.workunit.client.0.smithi060.stdout: "num_legacy_snapsets": 0, 2020-08-24T01:04:00.117 INFO:tasks.workunit.client.0.smithi060.stdout: "num_large_omap_objects": 0, 2020-08-24T01:04:00.117 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_manifest": 0, 2020-08-24T01:04:00.117 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_bytes": 0, 2020-08-24T01:04:00.117 INFO:tasks.workunit.client.0.smithi060.stdout: "num_omap_keys": 0, 2020-08-24T01:04:00.117 INFO:tasks.workunit.client.0.smithi060.stdout: "num_objects_repaired": 0 2020-08-24T01:04:00.118 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.118 INFO:tasks.workunit.client.0.smithi060.stdout: "store_stats": { 2020-08-24T01:04:00.118 INFO:tasks.workunit.client.0.smithi060.stdout: "total": 0, 2020-08-24T01:04:00.118 INFO:tasks.workunit.client.0.smithi060.stdout: "available": 0, 2020-08-24T01:04:00.118 INFO:tasks.workunit.client.0.smithi060.stdout: "internally_reserved": 0, 2020-08-24T01:04:00.118 INFO:tasks.workunit.client.0.smithi060.stdout: "allocated": 19857408, 2020-08-24T01:04:00.119 INFO:tasks.workunit.client.0.smithi060.stdout: "data_stored": 1241088, 2020-08-24T01:04:00.119 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed": 0, 2020-08-24T01:04:00.119 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_allocated": 0, 2020-08-24T01:04:00.119 INFO:tasks.workunit.client.0.smithi060.stdout: "data_compressed_original": 0, 2020-08-24T01:04:00.119 INFO:tasks.workunit.client.0.smithi060.stdout: "omap_allocated": 0, 2020-08-24T01:04:00.120 INFO:tasks.workunit.client.0.smithi060.stdout: "internal_metadata": 0 2020-08-24T01:04:00.120 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.120 INFO:tasks.workunit.client.0.smithi060.stdout: "log_size": 18, 2020-08-24T01:04:00.120 INFO:tasks.workunit.client.0.smithi060.stdout: "ondisk_log_size": 18, 2020-08-24T01:04:00.120 INFO:tasks.workunit.client.0.smithi060.stdout: "up": 3, 2020-08-24T01:04:00.121 INFO:tasks.workunit.client.0.smithi060.stdout: "acting": 3, 2020-08-24T01:04:00.121 INFO:tasks.workunit.client.0.smithi060.stdout: "num_store_stats": 3 2020-08-24T01:04:00.121 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.121 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.121 INFO:tasks.workunit.client.0.smithi060.stdout: "osd_stats": [ 2020-08-24T01:04:00.121 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.122 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:04:00.122 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 171798691850 2020-08-24T01:04:00.122 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.122 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.122 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:04:00.123 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 38654705685 2020-08-24T01:04:00.123 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.123 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.124 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:04:00.124 INFO:tasks.workunit.client.0.smithi060.stdout: "seq": 184683593737 2020-08-24T01:04:00.124 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.124 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.125 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_state": [ 2020-08-24T01:04:00.125 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.125 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "active+undersized+degraded", 2020-08-24T01:04:00.125 INFO:tasks.workunit.client.0.smithi060.stdout: "num": 1 2020-08-24T01:04:00.126 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.126 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.126 INFO:tasks.workunit.client.0.smithi060.stdout: "num_pg_by_osd": [ 2020-08-24T01:04:00.126 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.127 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2, 2020-08-24T01:04:00.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:04:00.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T01:04:00.127 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:04:00.128 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.128 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.128 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 2147483647, 2020-08-24T01:04:00.128 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:04:00.128 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T01:04:00.129 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:04:00.129 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.129 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.129 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 1, 2020-08-24T01:04:00.129 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 0, 2020-08-24T01:04:00.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 0, 2020-08-24T01:04:00.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:04:00.130 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.130 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.130 INFO:tasks.workunit.client.0.smithi060.stdout: "osd": 0, 2020-08-24T01:04:00.130 INFO:tasks.workunit.client.0.smithi060.stdout: "num_primary_pg": 1, 2020-08-24T01:04:00.131 INFO:tasks.workunit.client.0.smithi060.stdout: "num_acting_pg": 1, 2020-08-24T01:04:00.131 INFO:tasks.workunit.client.0.smithi060.stdout: "num_up_not_acting_pg": 0 2020-08-24T01:04:00.131 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.131 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.131 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": [ 2020-08-24T01:04:00.132 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:04:00.132 INFO:tasks.workunit.client.0.smithi060.stdout: "pool": 1, 2020-08-24T01:04:00.132 INFO:tasks.workunit.client.0.smithi060.stdout: "purged_snaps": {} 2020-08-24T01:04:00.132 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.132 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:04:00.133 INFO:tasks.workunit.client.0.smithi060.stdout: "servicemap": { 2020-08-24T01:04:00.133 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": 1, 2020-08-24T01:04:00.133 INFO:tasks.workunit.client.0.smithi060.stdout: "modified": "2020-08-24T01:01:12.875742+0000", 2020-08-24T01:04:00.133 INFO:tasks.workunit.client.0.smithi060.stdout: "services": {} 2020-08-24T01:04:00.133 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:04:00.133 INFO:tasks.workunit.client.0.smithi060.stdout: "paxos": { 2020-08-24T01:04:00.134 INFO:tasks.workunit.client.0.smithi060.stdout: "first_committed": 1, 2020-08-24T01:04:00.134 INFO:tasks.workunit.client.0.smithi060.stdout: "last_committed": 503, 2020-08-24T01:04:00.135 INFO:tasks.workunit.client.0.smithi060.stdout: "last_pn": 0, 2020-08-24T01:04:00.135 INFO:tasks.workunit.client.0.smithi060.stdout: "accepted_pn": 0 2020-08-24T01:04:00.135 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:04:00.135 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:04:00.136 INFO:tasks.workunit.client.0.smithi060.stderr:report 3733047823 2020-08-24T01:04:00.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-08-24T01:04:00.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-08-24T01:04:00.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:00.137 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery 2020-08-24T01:04:00.217 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:00.217 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-08-24T01:03:25.013805+0000 38'102 44:134 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:00.217 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:00.217 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:00.218 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:00.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:04:00.233 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-08-24T01:04:00.234 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T01:04:00.234 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T01:04:00.234 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:04:00.234 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:00.235 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:04:00.235 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.0.asok 2020-08-24T01:04:00.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-08-24T01:04:00.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.0.asok set_recovery_delay 100000 2020-08-24T01:04:00.327 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T01:04:00.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-08-24T01:04:00.336 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-08-24T01:04:00.336 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-08-24T01:04:00.336 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-08-24T01:04:00.338 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:04:00.338 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:00.340 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:04:00.340 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.2.asok 2020-08-24T01:04:00.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-08-24T01:04:00.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.2.asok set_recovery_delay 100000 2020-08-24T01:04:00.431 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T01:04:00.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-08-24T01:04:00.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:00.439 INFO:tasks.workunit.client.0.smithi060.stdout:revive divergent 1 2020-08-24T01:04:00.785 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:00.786 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-08-24T01:03:25.013805+0000 38'102 44:134 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:00.786 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:00.786 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:00.788 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:00.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-08-24T01:04:01.434 INFO:tasks.workunit.client.0.smithi060.stderr:noup is set 2020-08-24T01:04:01.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-08-24T01:04:01.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:04:01.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:04:01.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T01:04:01.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:04:01.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T01:04:01.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:04:01.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:04:01.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:04:01.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:04:01.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T01:04:01.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T01:04:01.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:04:01.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:04:01.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:04:01.460 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:04:01.460 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:04:01.460 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:04:01.460 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:04:01.461 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:01.461 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:04:01.461 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:04:01.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:04:01.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:04:01.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:04:01.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:04:01.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:04:01.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:04:01.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:01.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:04:01.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:04:01.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T01:04:01.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T01:04:01.464 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:01.464 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T01:04:01.464 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:04:01.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T01:04:01.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:04:01.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:04:01.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:04:01.493 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:01.488+0000 7fae87908d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:01.496 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:01.492+0000 7fae87908d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:01.498 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:01.492+0000 7fae87908d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:01.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-08-24T01:04:01.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-08-24T01:04:02.034 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:02.028+0000 7fae87908d80 -1 Falling back to public interface 2020-08-24T01:04:02.898 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:02.892+0000 7fae87908d80 -1 osd.1 28 log_to_monitors {default=true} 2020-08-24T01:04:06.840 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-08-24T01:04:41.013 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:41.014 INFO:tasks.workunit.client.0.smithi060.stdout:delay recovery divergent 2020-08-24T01:04:41.031 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:41.031 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-08-24T01:03:25.013805+0000 38'102 45:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:41.032 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:41.032 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:41.032 INFO:tasks.workunit.client.0.smithi060.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-08-24T01:04:41.032 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:04:41.033 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 40 up_thru 43 down_at 39 last_clean_interval [34,38) [v2:127.0.0.1:6802/44430,v1:127.0.0.1:6803/44430] [v2:127.0.0.1:6804/44430,v1:127.0.0.1:6805/44430] exists,up 2590757d-b008-4deb-b066-fccba9726114 2020-08-24T01:04:41.033 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:04:41.033 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/46017,v1:127.0.0.1:6819/46017] [v2:127.0.0.1:6820/46017,v1:127.0.0.1:6821/46017] exists,up eb457a2e-e0bb-468b-864e-4386438ddbc4 2020-08-24T01:04:41.033 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:04:41.033 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 43 up_thru 0 down_at 42 last_clean_interval [35,38) [v2:127.0.0.1:6810/44662,v1:127.0.0.1:6811/44662] [v2:127.0.0.1:6812/44662,v1:127.0.0.1:6813/44662] exists,up 813ed60f-f06b-4b40-a01a-9681d9e7dd5d 2020-08-24T01:04:41.034 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:41.035 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-08-24T01:03:25.013805+0000 38'102 45:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:41.035 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:41.035 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:41.035 INFO:tasks.workunit.client.0.smithi060.stdout:wait for peering 2020-08-24T01:04:41.035 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:41.036 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 0 101 0 0 62471 0 0 18 18 active+undersized+degraded 2020-08-24T01:03:25.013805+0000 38'102 45:136 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:41.036 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:41.036 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:41.036 INFO:tasks.workunit.client.0.smithi060.stdout:killing divergent 1 2020-08-24T01:04:41.036 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:41.037 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 4 4 0 0 62471 0 0 12 12 active+recovering+degraded+wait 2020-08-24T01:04:08.962559+0000 38'102 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:41.037 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:41.037 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:41.037 INFO:tasks.workunit.client.0.smithi060.stdout:reviving divergent 1 2020-08-24T01:04:41.037 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:41.038 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 101 4 4 0 0 62471 0 0 12 12 active+recovering+degraded+wait 2020-08-24T01:04:08.962559+0000 38'102 49:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:41.038 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:41.038 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:41.038 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:04:41.038 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:04:41.039 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/46017,v1:127.0.0.1:6819/46017] [v2:127.0.0.1:6820/46017,v1:127.0.0.1:6821/46017] exists,up eb457a2e-e0bb-468b-864e-4386438ddbc4 2020-08-24T01:04:41.039 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:41.039 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-08-24T01:04:41.039 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:04:41.040 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:04:41.040 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:04:41.040 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:41.040 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:04:41.040 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.11819/ceph-osd.1.asok 2020-08-24T01:04:41.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-08-24T01:04:41.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.11819/ceph-osd.1.asok set_recovery_delay 100000 2020-08-24T01:04:41.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-08-24T01:04:41.041 INFO:tasks.workunit.client.0.smithi060.stderr:noup is unset 2020-08-24T01:04:41.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-08-24T01:04:41.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:04:41.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:04:41.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:04:41.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:04:41.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:41.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:04:41.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:41.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:04:41.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:04:41.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:04:41.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:04:41.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-08-24T01:04:41.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:04:41.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:04:41.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:04:41.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:04:41.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:41.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:04:41.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:41.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:04:41.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:04:41.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:04:41.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:04:41.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-08-24T01:04:41.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:04:41.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:04:41.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:04:41.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:04:41.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:41.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:04:41.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:41.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:04:41.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:04:41.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:04:41.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:04:41.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:41.049 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:41.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-08-24T01:04:41.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:41.050 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:41.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-08-24T01:04:41.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-08-24T01:04:41.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:41.050 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:41.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-08-24T01:04:41.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:04:41.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:04:41.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:04:41.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:04:41.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:04:41.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:04:41.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-08-24T01:04:41.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:41.052 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:41.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-08-24T01:04:41.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-08-24T01:04:41.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:04:41.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T01:04:41.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:04:41.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-08-24T01:04:41.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-08-24T01:04:41.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:04:41.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:04:41.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:04:41.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-08-24T01:04:41.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-08-24T01:04:41.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:04:41.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:04:41.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-08-24T01:04:41.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:04:41.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:04:41.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:04:41.057 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:04:41.057 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:41.057 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:04:41.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:04:41.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' 2020-08-24T01:04:41.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:04:41.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-08-24T01:04:41.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-08-24T01:04:41.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:04:41.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:04:41.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:41.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:04:41.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:04:41.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-08-24T01:04:41.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T01:04:41.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=a4b7fd1e-bb41-4214-9732-51a9ef4bc5a0 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.11819/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:41.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-08-24T01:04:41.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T01:04:41.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:04:41.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:04:41.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:04:41.061 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:22.024+0000 7f718fdd5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:41.061 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:22.024+0000 7f718fdd5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:41.061 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:22.028+0000 7f718fdd5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:41.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-08-24T01:04:41.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:04:41.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:04:41.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:04:41.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:04:41.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:41.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:04:41.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:41.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:04:41.063 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:22.568+0000 7f718fdd5d80 -1 Falling back to public interface 2020-08-24T01:04:41.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:04:41.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:04:41.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:04:41.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-08-24T01:04:41.064 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:23.432+0000 7f718fdd5d80 -1 osd.1 49 log_to_monitors {default=true} 2020-08-24T01:04:42.739 INFO:tasks.workunit.client.0.smithi060.stdout:allowing recovery 2020-08-24T01:04:42.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-08-24T01:04:42.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:43.106 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:43.107 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 102 0 0 0 0 63088 0 0 13 13 active+clean 2020-08-24T01:04:25.047551+0000 49'103 52:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:43.107 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:43.107 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:43.108 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:43.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-08-24T01:04:43.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-08-24T01:04:43.269 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T01:04:43.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-08-24T01:04:43.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-08-24T01:04:43.427 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T01:04:43.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-08-24T01:04:43.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-08-24T01:04:43.595 INFO:tasks.workunit.client.0.smithi060.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-08-24T01:04:43.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-08-24T01:04:43.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-08-24T01:04:43.604 INFO:tasks.workunit.client.0.smithi060.stdout:reading divergent objects 2020-08-24T01:04:43.953 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:04:43.955 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 102 0 0 0 0 63088 0 0 13 13 active+clean 2020-08-24T01:04:25.047551+0000 49'103 52:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-08-24T01:01:36.306156+0000 0'0 2020-08-24T01:01:36.306156+0000 0 2020-08-24T01:04:43.955 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:04:43.955 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:04:43.956 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:04:43.976 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-08-24T01:04:43.977 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-08-24T01:04:43.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:04:43.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-08-24T01:04:44.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:04:44.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-08-24T01:04:44.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:04:44.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-08-24T01:04:44.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-08-24T01:04:44.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-08-24T01:04:44.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-08-24T01:04:44.175 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-08-24T01:04:44.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T01:04:44.182 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.796+0000 7fae66545700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-08-24T01:02:10.952966+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.182 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.796+0000 7fae66545700 20 _merge_object_divergent_entries: keeping 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-08-24T01:02:10.952966+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.183 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.796+0000 7fae66545700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:04:44.183 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.796+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 27'99 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-08-24T01:04:44.183 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.796+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-08-24T01:04:44.183 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.796+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-08-24T01:04:44.184 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-08-24T01:02:18.253346+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.184 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 20 _merge_object_divergent_entries: keeping 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-08-24T01:02:18.253346+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.184 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:04:44.184 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 27'99 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-08-24T01:04:44.184 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-08-24T01:04:44.185 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-08-24T01:04:44.185 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 28'104 2020-08-24T01:04:44.185 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4557.0:1 2020-08-24T01:02:18.253346+0000 0 ObjectCleanRegions clean_offsets: [617~18446744073709550998], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.185 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-08-24T01:04:44.185 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-08-24T01:02:17.062616+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0,27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-08-24T01:02:18.064714+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.186 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4548.0:1 2020-08-24T01:02:17.062616+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.186 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 20 _merge_object_divergent_entries: keeping 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-08-24T01:02:18.064714+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.187 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-08-24T01:04:44.187 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'102 2020-08-24T01:04:44.187 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-08-24T01:04:44.187 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-08-24T01:04:44.187 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'104 2020-08-24T01:04:44.188 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4554.0:1 2020-08-24T01:02:18.064714+0000 0 ObjectCleanRegions clean_offsets: [771~18446744073709550844], clean_omap: 1, new_object: 0 2020-08-24T01:04:44.188 INFO:tasks.workunit.client.0.smithi060.stdout:td/divergent-priors/osd.1.log:2020-08-24T01:04:08.812+0000 7fae66545700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-08-24T01:04:44.188 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-08-24T01:04:44.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.2.log td/divergent-priors/osd.1.log td/divergent-priors/osd.0.log 2020-08-24T01:04:44.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-08-24T01:04:44.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-08-24T01:04:44.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-08-24T01:04:44.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-08-24T01:04:44.205 INFO:tasks.workunit.client.0.smithi060.stdout:success 2020-08-24T01:04:44.708 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' does not exist 2020-08-24T01:04:44.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-08-24T01:04:44.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:04:44.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:04:44.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:04:44.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:04:44.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:04:45.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:04:45.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-08-24T01:04:45.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T01:04:45.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:04:45.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T01:04:45.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:04:45.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:04:45.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:04:45.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:04:45.039 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:04:45.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:04:45.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:04:45.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:04:45.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:04:45.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:04:45.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:04:45.054 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:04:45.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:04:45.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:04:45.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:04:45.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:04:45.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:04:45.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:04:45.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T01:04:45.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:04:45.079 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:45.079 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:04:45.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T01:04:45.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:04:45.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:04:45.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: code=0 2020-08-24T01:04:45.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: teardown td/divergent-priors 0 2020-08-24T01:04:45.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-08-24T01:04:45.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-08-24T01:04:45.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-08-24T01:04:45.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:04:45.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:04:45.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:04:45.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:04:45.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:04:45.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:04:45.087 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:04:45.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:04:45.088 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:04:45.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:04:45.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:04:45.091 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:04:45.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:04:45.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:04:45.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:04:45.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:04:45.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:04:45.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-08-24T01:04:45.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-08-24T01:04:45.110 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:04:45.110 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:45.111 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.11819 2020-08-24T01:04:45.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.11819 2020-08-24T01:04:45.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:04:45.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:04:45.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: return 0 2020-08-24T01:04:45.113 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-08-24T01:04:45.114 INFO:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-08-24T01:04:45.165 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-08-24T01:04:45.166 INFO:teuthology.orchestra.run.smithi060:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a941e55043ccee6d35a8717bec55ff7581e67d8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-08-24T01:04:45.228 INFO:tasks.workunit.client.0.smithi060.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-08-24T01:04:45.239 INFO:tasks.workunit.client.0.smithi060.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-08-24T01:04:45.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T01:04:45.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T01:04:45.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PYTHONWARNINGS=ignore 2020-08-24T01:04:45.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PYTHONWARNINGS=ignore 2020-08-24T01:04:45.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export CEPH_CONF=/dev/null 2020-08-24T01:04:45.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: CEPH_CONF=/dev/null 2020-08-24T01:04:45.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: unset CEPH_ARGS 2020-08-24T01:04:45.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: local code 2020-08-24T01:04:45.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: run td/ec-error-rollforward 2020-08-24T01:04:45.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-08-24T01:04:45.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-08-24T01:04:45.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-08-24T01:04:45.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-08-24T01:04:45.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-08-24T01:04:45.246 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-08-24T01:04:45.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none ' 2020-08-24T01:04:45.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-08-24T01:04:45.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-08-24T01:04:45.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-08-24T01:04:45.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-08-24T01:04:45.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-08-24T01:04:45.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-08-24T01:04:45.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-08-24T01:04:45.248 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-08-24T01:04:45.249 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-08-24T01:04:45.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-08-24T01:04:45.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-08-24T01:04:45.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-08-24T01:04:45.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-08-24T01:04:45.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-08-24T01:04:45.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-08-24T01:04:45.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:04:45.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-08-24T01:04:45.250 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:04:45.250 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:04:45.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:04:45.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:04:45.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:04:45.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:04:45.253 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:04:45.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:04:45.253 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:04:45.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:04:45.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:04:45.255 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:04:45.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:04:45.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:04:45.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:04:45.256 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:04:45.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:04:45.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:04:45.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-08-24T01:04:45.260 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:04:45.260 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:45.260 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:45.261 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46909 2020-08-24T01:04:45.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:04:45.265 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:04:45.265 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-08-24T01:04:45.265 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:04:45.266 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:45.266 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:45.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.46909 2020-08-24T01:04:45.267 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:04:45.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-08-24T01:04:45.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-08-24T01:04:45.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:04:45.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-08-24T01:04:45.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-08-24T01:04:45.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-08-24T01:04:45.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-08-24T01:04:45.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-08-24T01:04:45.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:04:45.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:04:45.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:04:45.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-08-24T01:04:45.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-08-24T01:04:45.472 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:04:45.472 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:04:45.473 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:04:45.473 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:04:45.473 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:45.474 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:45.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:45.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:04:45.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:04:45.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:04:45.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:04:45.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:04:45.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:04:45.655 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:04:45.655 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:04:45.655 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:04:45.663 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:04:45.663 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:04:45.664 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:45.664 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:45.664 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46909/ceph-mon.a.asok 2020-08-24T01:04:45.664 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:04:45.665 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.46909/ceph-mon.a.asok config get fsid 2020-08-24T01:04:45.750 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:04:45.750 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:04:45.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:04:45.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:04:45.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:04:45.752 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:04:45.752 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:04:45.753 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:04:45.753 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:04:45.753 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:45.753 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:45.753 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.46909/ceph-mon.a.asok 2020-08-24T01:04:45.754 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:04:45.754 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.46909/ceph-mon.a.asok config get mon_host 2020-08-24T01:04:45.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-08-24T01:04:45.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-08-24T01:04:45.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:04:45.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:04:45.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:04:45.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-08-24T01:04:45.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:04:46.353 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:04:46.354 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:04:46.354 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:04:46.355 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:04:46.355 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:46.355 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:46.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:46.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:04:46.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:04:46.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-08-24T01:04:46.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-08-24T01:04:46.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:04:46.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:04:46.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:04:46.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-08-24T01:04:46.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:04:46.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:04:46.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:04:46.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:04:46.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-08-24T01:04:46.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-08-24T01:04:46.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:04:46.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:04:46.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:04:46.400 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:04:46.400 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:04:46.400 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:04:46.406 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:04:46.409 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:46.410 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:46.410 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:46.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:46.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:04:46.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:04:46.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:04:46.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:04:46.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:04:46.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:04:46.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:04:46.424 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:46.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:04:46.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:04:46.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-08-24T01:04:46.426 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:04:46.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=071220a6-550f-4924-97bc-aca8aff65fb9 2020-08-24T01:04:46.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 071220a6-550f-4924-97bc-aca8aff65fb9' 2020-08-24T01:04:46.427 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:04:46.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAuEkNfMUgKGBAAzsYx1iFizRs76K13F3b3vQ== 2020-08-24T01:04:46.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAuEkNfMUgKGBAAzsYx1iFizRs76K13F3b3vQ=="}' 2020-08-24T01:04:46.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 071220a6-550f-4924-97bc-aca8aff65fb9 -i td/ec-error-rollforward/0/new.json 2020-08-24T01:04:46.428 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 071220a6-550f-4924-97bc-aca8aff65fb9 2020-08-24T01:04:46.733 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:04:46.743 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-08-24T01:04:46.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAuEkNfMUgKGBAAzsYx1iFizRs76K13F3b3vQ== --osd-uuid 071220a6-550f-4924-97bc-aca8aff65fb9 2020-08-24T01:04:46.772 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:46.768+0000 7f8f60dd7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:46.774 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:46.768+0000 7f8f60dd7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:46.775 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:46.768+0000 7f8f60dd7d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:46.775 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:46.772+0000 7f8f60dd7d80 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-08-24T01:04:46.776 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:46.772+0000 7f8f60dd7d80 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-08-24T01:04:46.776 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:46.772+0000 7f8f60dd7d80 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-08-24T01:04:46.842 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:46.836+0000 7f8f60dd7d80 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-08-24T01:04:47.235 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:47.228+0000 7f8f60dd7d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:04:49.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-08-24T01:04:49.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:04:49.976 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:04:49.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:04:49.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:04:50.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:04:50.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:50.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:04:50.431 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:04:50.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:04:50.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:04:50.454 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:50.448+0000 7f0c896bfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:50.456 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:50.452+0000 7f0c896bfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:50.457 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:50.452+0000 7f0c896bfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:50.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:04:50.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:04:50.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:04:50.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:04:50.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:04:50.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:50.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:04:50.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:50.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:04:50.781 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:04:50.997 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:50.992+0000 7f0c896bfd80 -1 Falling back to public interface 2020-08-24T01:04:51.154 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:04:51.793 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:51.780+0000 7f0c896bfd80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:04:52.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:04:52.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:52.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:04:52.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:52.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:04:52.158 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:04:52.541 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/47331,v1:127.0.0.1:6803/47331] [v2:127.0.0.1:6804/47331,v1:127.0.0.1:6805/47331] exists,up 071220a6-550f-4924-97bc-aca8aff65fb9 2020-08-24T01:04:52.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:04:52.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:04:52.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:04:52.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-08-24T01:04:52.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-08-24T01:04:52.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:04:52.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:04:52.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:04:52.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-08-24T01:04:52.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:04:52.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:04:52.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:04:52.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:04:52.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-08-24T01:04:52.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-08-24T01:04:52.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:04:52.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:04:52.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:04:52.545 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:04:52.546 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:04:52.546 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:04:52.546 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:04:52.546 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:52.547 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:52.547 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:52.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:52.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:04:52.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:04:52.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:04:52.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:04:52.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:04:52.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:04:52.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:04:52.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:52.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:04:52.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:04:52.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-08-24T01:04:52.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:04:52.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=45645808-e5ec-430c-bc32-23a899a8c81c 2020-08-24T01:04:52.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 45645808-e5ec-430c-bc32-23a899a8c81c' 2020-08-24T01:04:52.552 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:04:52.552 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 45645808-e5ec-430c-bc32-23a899a8c81c 2020-08-24T01:04:52.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA0EkNfh4dhIRAAUgbQaKZvOilp4jyl9PPEcw== 2020-08-24T01:04:52.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA0EkNfh4dhIRAAUgbQaKZvOilp4jyl9PPEcw=="}' 2020-08-24T01:04:52.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 45645808-e5ec-430c-bc32-23a899a8c81c -i td/ec-error-rollforward/1/new.json 2020-08-24T01:04:53.129 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:04:53.145 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-08-24T01:04:53.146 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA0EkNfh4dhIRAAUgbQaKZvOilp4jyl9PPEcw== --osd-uuid 45645808-e5ec-430c-bc32-23a899a8c81c 2020-08-24T01:04:53.175 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.168+0000 7fb2fefdbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:53.200 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.196+0000 7fb2fefdbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:53.201 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.196+0000 7fb2fefdbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:53.202 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.196+0000 7fb2fefdbd80 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-08-24T01:04:53.202 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.196+0000 7fb2fefdbd80 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-08-24T01:04:53.203 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.196+0000 7fb2fefdbd80 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-08-24T01:04:53.287 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.280+0000 7fb2fefdbd80 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-08-24T01:04:53.667 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:53.664+0000 7fb2fefdbd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:04:56.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-08-24T01:04:56.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:04:56.414 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:04:56.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:04:56.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:04:56.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:04:56.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:56.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:04:56.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:04:56.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:04:56.881 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:04:56.903 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:56.896+0000 7f9fa3d83d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:56.905 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:56.900+0000 7f9fa3d83d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:56.906 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:56.900+0000 7f9fa3d83d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:57.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:04:57.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:04:57.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:04:57.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:04:57.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:04:57.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:57.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:04:57.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:04:57.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:57.234 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:04:57.441 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:57.436+0000 7f9fa3d83d80 -1 Falling back to public interface 2020-08-24T01:04:57.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:04:58.227 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:58.224+0000 7f9fa3d83d80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:04:58.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:04:58.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:04:58.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:04:58.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:04:58.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:04:58.612 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:04:58.975 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/47686,v1:127.0.0.1:6811/47686] [v2:127.0.0.1:6812/47686,v1:127.0.0.1:6813/47686] exists,up 45645808-e5ec-430c-bc32-23a899a8c81c 2020-08-24T01:04:58.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:04:58.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:04:58.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:04:58.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-08-24T01:04:58.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-08-24T01:04:58.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:04:58.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T01:04:58.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:04:58.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-08-24T01:04:58.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:04:58.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:04:58.994 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:04:59.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:04:59.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-08-24T01:04:59.002 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-08-24T01:04:59.002 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:04:59.002 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:04:59.003 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:04:59.014 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:04:59.015 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:04:59.015 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:04:59.015 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:04:59.015 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:04:59.015 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:04:59.016 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:59.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:04:59.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:04:59.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:04:59.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:04:59.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:04:59.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:04:59.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:04:59.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:04:59.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:04:59.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:04:59.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:04:59.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-08-24T01:04:59.019 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:04:59.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=20a5af19-cbc0-4e86-ac02-1ee261e73ef0 2020-08-24T01:04:59.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 20a5af19-cbc0-4e86-ac02-1ee261e73ef0' 2020-08-24T01:04:59.020 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:04:59.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA7EkNfqewNABAAqC3GqIemZCdmW8PKqT9TXA== 2020-08-24T01:04:59.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA7EkNfqewNABAAqC3GqIemZCdmW8PKqT9TXA=="}' 2020-08-24T01:04:59.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 20a5af19-cbc0-4e86-ac02-1ee261e73ef0 -i td/ec-error-rollforward/2/new.json 2020-08-24T01:04:59.021 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 20a5af19-cbc0-4e86-ac02-1ee261e73ef0 2020-08-24T01:04:59.587 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:04:59.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-08-24T01:04:59.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA7EkNfqewNABAAqC3GqIemZCdmW8PKqT9TXA== --osd-uuid 20a5af19-cbc0-4e86-ac02-1ee261e73ef0 2020-08-24T01:04:59.627 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:59.624+0000 7f8547db3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:59.630 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:59.624+0000 7f8547db3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:59.632 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:59.628+0000 7f8547db3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:04:59.632 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:59.628+0000 7f8547db3d80 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-08-24T01:04:59.633 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:59.628+0000 7f8547db3d80 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-08-24T01:04:59.633 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:59.628+0000 7f8547db3d80 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-08-24T01:04:59.681 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:04:59.676+0000 7f8547db3d80 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-08-24T01:05:00.142 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:00.136+0000 7f8547db3d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:05:02.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-08-24T01:05:29.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:05:29.488 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T01:05:29.488 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:05:29.489 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:29.489 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:05:29.489 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/48044,v1:127.0.0.1:6819/48044] [v2:127.0.0.1:6820/48044,v1:127.0.0.1:6821/48044] exists,up 20a5af19-cbc0-4e86-ac02-1ee261e73ef0 2020-08-24T01:05:29.489 INFO:tasks.workunit.client.0.smithi060.stdout:add osd3 8ade8164-010c-4702-9d2c-016659e4315b 2020-08-24T01:05:29.489 INFO:tasks.workunit.client.0.smithi060.stdout:3 2020-08-24T01:05:29.490 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd3 key to auth repository 2020-08-24T01:05:29.490 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.3 2020-08-24T01:05:29.490 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:29.490 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:05:29.490 INFO:tasks.workunit.client.0.smithi060.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/48399,v1:127.0.0.1:6827/48399] [v2:127.0.0.1:6828/48399,v1:127.0.0.1:6829/48399] exists,up 8ade8164-010c-4702-9d2c-016659e4315b 2020-08-24T01:05:29.491 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:05:29.491 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:29.491 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/47331,v1:127.0.0.1:6803/47331] [v2:127.0.0.1:6804/47331,v1:127.0.0.1:6805/47331] exists,up 071220a6-550f-4924-97bc-aca8aff65fb9 2020-08-24T01:05:29.491 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:05:29.491 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:29.492 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 24 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/47686,v1:127.0.0.1:6811/47686] [v2:127.0.0.1:6812/47686,v1:127.0.0.1:6813/47686] exists,up 45645808-e5ec-430c-bc32-23a899a8c81c 2020-08-24T01:05:29.492 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:05:29.492 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:29.492 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:05:29.492 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:05:29.493 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 33 up_thru 0 down_at 29 last_clean_interval [15,25) [v2:127.0.0.1:6818/49063,v1:127.0.0.1:6819/49063] [v2:127.0.0.1:6820/49063,v1:127.0.0.1:6821/49063] exists,up 20a5af19-cbc0-4e86-ac02-1ee261e73ef0 2020-08-24T01:05:29.493 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.3 2020-08-24T01:05:29.493 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:29.493 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:05:29.493 INFO:tasks.workunit.client.0.smithi060.stdout:osd.3 up in weight 1 up_from 35 up_thru 0 down_at 29 last_clean_interval [20,26) [v2:127.0.0.1:6826/49368,v1:127.0.0.1:6827/49368] [v2:127.0.0.1:6828/49368,v1:127.0.0.1:6829/49368] exists,up 8ade8164-010c-4702-9d2c-016659e4315b 2020-08-24T01:05:29.493 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 120259084292 2020-08-24T01:05:29.494 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 133143986180 2020-08-24T01:05:29.494 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 141733920771 2020-08-24T01:05:29.494 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 150323855363 2020-08-24T01:05:29.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T01:05:29.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:05:29.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T01:05:29.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:05:29.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:05:29.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:05:29.496 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:03.260+0000 7f3298fe1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.496 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:03.264+0000 7f3298fe1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.496 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:03.264+0000 7f3298fe1d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T01:05:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:05:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:29.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:29.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:05:29.498 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:03.808+0000 7f3298fe1d80 -1 Falling back to public interface 2020-08-24T01:05:29.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:05:29.499 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:04.640+0000 7f3298fe1d80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T01:05:29.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:05:29.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:05:29.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:05:29.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:29.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:29.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:29.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-08-24T01:05:29.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-08-24T01:05:29.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:05:29.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-08-24T01:05:29.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:05:29.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-08-24T01:05:29.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:05:29.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:05:29.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:05:29.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:05:29.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-08-24T01:05:29.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-08-24T01:05:29.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:05:29.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:05:29.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:05:29.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:05:29.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:29.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:29.504 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:29.504 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.505 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:05:29.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:05:29.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:05:29.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:05:29.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:05:29.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:05:29.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:05:29.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:05:29.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:05:29.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:05:29.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-08-24T01:05:29.508 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:05:29.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8ade8164-010c-4702-9d2c-016659e4315b 2020-08-24T01:05:29.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 8ade8164-010c-4702-9d2c-016659e4315b' 2020-08-24T01:05:29.508 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:05:29.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBBEkNfQ7rDFBAA8zuuzB0HQl44Fj8TjL//Kw== 2020-08-24T01:05:29.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBBEkNfQ7rDFBAA8zuuzB0HQl44Fj8TjL//Kw=="}' 2020-08-24T01:05:29.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8ade8164-010c-4702-9d2c-016659e4315b -i td/ec-error-rollforward/3/new.json 2020-08-24T01:05:29.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-08-24T01:05:29.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBBEkNfQ7rDFBAA8zuuzB0HQl44Fj8TjL//Kw== --osd-uuid 8ade8164-010c-4702-9d2c-016659e4315b 2020-08-24T01:05:29.509 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:05.948+0000 7fcf07ee2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.510 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:05.952+0000 7fcf07ee2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.510 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:05.956+0000 7fcf07ee2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.510 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:05.956+0000 7fcf07ee2d80 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-08-24T01:05:29.510 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:05.956+0000 7fcf07ee2d80 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-08-24T01:05:29.510 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:05.956+0000 7fcf07ee2d80 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-08-24T01:05:29.511 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:06.028+0000 7fcf07ee2d80 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-08-24T01:05:29.511 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:06.628+0000 7fcf07ee2d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:05:29.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-08-24T01:05:29.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:05:29.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-08-24T01:05:29.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:05:29.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-08-24T01:05:29.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:05:29.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:05:29.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:05:29.513 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:09.792+0000 7f8c41fafd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.513 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:09.792+0000 7f8c41fafd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.513 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:09.796+0000 7f8c41fafd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-08-24T01:05:29.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:29.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-08-24T01:05:29.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:29.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:29.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:29.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:05:29.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.515 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:10.332+0000 7f8c41fafd80 -1 Falling back to public interface 2020-08-24T01:05:29.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:05:29.516 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:11.096+0000 7f8c41fafd80 -1 osd.3 0 log_to_monitors {default=true} 2020-08-24T01:05:29.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:05:29.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:05:29.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:05:29.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:29.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:29.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:29.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-08-24T01:05:29.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-08-24T01:05:29.518 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'ec' already exists 2020-08-24T01:05:29.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-08-24T01:05:29.518 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-08-24T01:05:29.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 48044 2020-08-24T01:05:29.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=48684 2020-08-24T01:05:29.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-08-24T01:05:29.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-08-24T01:05:29.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 48707' 2020-08-24T01:05:29.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-08-24T01:05:29.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 48708' 2020-08-24T01:05:29.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-08-24T01:05:29.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 48709' 2020-08-24T01:05:29.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-08-24T01:05:29.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-08-24T01:05:29.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-08-24T01:05:29.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:05:29.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:05:29.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:05:29.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:05:29.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:05:29.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:05:29.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 48684 48707 48708 48709 2020-08-24T01:05:29.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-08-24T01:05:29.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-08-24T01:05:29.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-08-24T01:05:29.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:05:29.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-08-24T01:05:29.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:05:29.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-08-24T01:05:29.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:05:29.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:05:29.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:05:29.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:05:29.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-08-24T01:05:29.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-08-24T01:05:29.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:05:29.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:05:29.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:05:29.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:05:29.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:29.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:29.526 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:29.526 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.527 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:05:29.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:05:29.527 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:05:29.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:05:29.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:05:29.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:05:29.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:05:29.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:05:29.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-08-24T01:05:29.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-08-24T01:05:29.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.529 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-08-24T01:05:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-08-24T01:05:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:05:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:05:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:05:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:17.376+0000 7f6bfc516d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.531 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:17.376+0000 7f6bfc516d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.531 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:17.380+0000 7f6bfc516d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-08-24T01:05:29.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:29.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:05:29.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:29.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:29.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:29.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:05:29.533 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:17.920+0000 7f6bfc516d80 -1 Falling back to public interface 2020-08-24T01:05:29.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:29.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:29.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:29.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-08-24T01:05:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-08-24T01:05:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:05:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-08-24T01:05:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:05:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-08-24T01:05:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:05:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:05:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:05:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:05:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-08-24T01:05:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-08-24T01:05:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:05:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:05:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:05:29.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:05:29.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:29.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:29.537 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:29.537 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.538 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:05:29.538 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:05:29.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:05:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:05:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:05:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:05:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:05:29.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:05:29.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-08-24T01:05:29.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-08-24T01:05:29.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-08-24T01:05:29.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-08-24T01:05:29.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:05:29.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:05:29.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:05:29.541 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.112+0000 7efc49bcbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.541 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.116+0000 7efc49bcbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.116+0000 7efc49bcbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-08-24T01:05:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:05:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:05:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.660+0000 7efc49bcbd80 -1 Falling back to public interface 2020-08-24T01:05:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.812+0000 7f6bfc516d80 -1 osd.0 26 log_to_monitors {default=true} 2020-08-24T01:05:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-08-24T01:05:29.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-08-24T01:05:29.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:05:29.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-08-24T01:05:29.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:05:29.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-08-24T01:05:29.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:05:29.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:05:29.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:05:29.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:05:29.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-08-24T01:05:29.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-08-24T01:05:29.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:05:29.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:05:29.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:05:29.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:05:29.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:29.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:29.548 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:29.548 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.549 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:05:29.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:05:29.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:05:29.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:05:29.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:05:29.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:05:29.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:05:29.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:05:29.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-08-24T01:05:29.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-08-24T01:05:29.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-08-24T01:05:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-08-24T01:05:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:05:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:05:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:05:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.868+0000 7fa1d8364d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.868+0000 7fa1d8364d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:18.872+0000 7fa1d8364d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-08-24T01:05:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:05:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:05:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.555 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:19.416+0000 7fa1d8364d80 -1 Falling back to public interface 2020-08-24T01:05:29.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:05:29.555 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:19.808+0000 7efc49bcbd80 -1 osd.1 26 log_to_monitors {default=true} 2020-08-24T01:05:29.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:05:29.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:05:29.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:05:29.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.556 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:20.880+0000 7fa1d8364d80 -1 osd.2 25 log_to_monitors {default=true} 2020-08-24T01:05:29.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:05:29.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:05:29.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:05:29.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:05:29.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:29.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:29.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:29.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-08-24T01:05:29.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-08-24T01:05:29.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-08-24T01:05:29.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-08-24T01:05:29.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-08-24T01:05:29.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-08-24T01:05:29.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-08-24T01:05:29.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:05:29.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:05:29.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:05:29.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-08-24T01:05:29.560 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-08-24T01:05:29.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-08-24T01:05:29.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-08-24T01:05:29.561 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-08-24T01:05:29.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-08-24T01:05:29.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:29.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:29.562 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:29.562 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.562 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:05:29.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' 2020-08-24T01:05:29.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:05:29.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-08-24T01:05:29.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-08-24T01:05:29.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:05:29.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:05:29.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-08-24T01:05:29.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-08-24T01:05:29.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-08-24T01:05:29.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-08-24T01:05:29.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=632362ac-2fba-4fdb-a7e7-b01aa5ee2ec8 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.46909/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:29.565 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-08-24T01:05:29.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-08-24T01:05:29.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-08-24T01:05:29.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-08-24T01:05:29.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-08-24T01:05:29.566 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:22.432+0000 7efcc155bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.566 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:22.436+0000 7efcc155bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.567 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:22.436+0000 7efcc155bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:29.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-08-24T01:05:29.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:29.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-08-24T01:05:29.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:29.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:29.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:29.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:05:29.569 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:22.968+0000 7efcc155bd80 -1 Falling back to public interface 2020-08-24T01:05:29.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:05:29.569 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:23.964+0000 7efcc155bd80 -1 osd.3 26 log_to_monitors {default=true} 2020-08-24T01:05:29.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:05:29.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:29.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:05:29.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:29.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:05:29.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:29.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:29.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:29.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-08-24T01:05:29.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:05:29.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:05:29.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:05:29.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:05:29.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:05:29.572 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:05:29.572 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:05:29.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:05:29.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:05:29.573 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:05:29.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:05:29.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:05:29.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:05:29.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:05:29.573 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:05:29.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:05:29.574 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:05:29.574 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:05:29.574 INFO:tasks.workunit.client.0.smithi060.stderr:3' 2020-08-24T01:05:29.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:05:29.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:05:29.575 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:05:29.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=120259084292 2020-08-24T01:05:29.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 120259084292 2020-08-24T01:05:29.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-120259084292' 2020-08-24T01:05:29.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:05:29.576 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:05:29.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=133143986180 2020-08-24T01:05:29.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 133143986180 2020-08-24T01:05:29.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-120259084292 1-133143986180' 2020-08-24T01:05:29.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:05:29.577 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:05:29.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=141733920771 2020-08-24T01:05:29.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 141733920771 2020-08-24T01:05:29.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-120259084292 1-133143986180 2-141733920771' 2020-08-24T01:05:29.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:05:29.578 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:05:29.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-08-24T01:05:29.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 150323855363 2020-08-24T01:05:29.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-120259084292 1-133143986180 2-141733920771 3-150323855363' 2020-08-24T01:05:29.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:05:29.579 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-120259084292 2020-08-24T01:05:29.579 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:05:29.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:05:29.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-120259084292 2020-08-24T01:05:29.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:05:29.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=120259084292 2020-08-24T01:05:29.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 120259084292' 2020-08-24T01:05:29.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:05:29.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 120259084292 -lt 120259084292 2020-08-24T01:05:29.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:05:29.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-133143986180 2020-08-24T01:05:29.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:05:29.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:05:29.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-133143986180 2020-08-24T01:05:29.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:05:29.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=133143986180 2020-08-24T01:05:29.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 133143986180' 2020-08-24T01:05:29.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:05:29.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 133143986180 -lt 133143986180 2020-08-24T01:05:29.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:05:29.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-141733920771 2020-08-24T01:05:29.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:05:29.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:05:29.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-141733920771 2020-08-24T01:05:29.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:05:29.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=141733920771 2020-08-24T01:05:29.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 141733920771' 2020-08-24T01:05:29.584 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:05:29.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 141733920771 -lt 141733920771 2020-08-24T01:05:29.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:05:29.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-150323855363 2020-08-24T01:05:29.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:05:29.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:05:29.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-150323855363 2020-08-24T01:05:29.586 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:05:29.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=150323855363 2020-08-24T01:05:29.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 150323855363' 2020-08-24T01:05:29.586 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:05:29.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 150323855363 -lt 150323855363 2020-08-24T01:05:29.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:05:29.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:05:29.587 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:05:29.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:05:29.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:05:29.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:05:29.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:05:29.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:05:29.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:05:29.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:05:29.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:05:29.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:05:29.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:05:29.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:05:29.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:05:29.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T01:05:29.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:05:29.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:05:29.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-08-24T01:05:29.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-08-24T01:05:29.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:05:29.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-08-24T01:05:29.591 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:05:29.591 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:05:29.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:05:29.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:05:29.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:05:29.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:05:29.592 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:05:29.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:05:29.593 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:05:29.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:05:29.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:05:29.593 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:05:29.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:05:29.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:05:29.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:05:29.594 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:05:29.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:05:29.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:05:29.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-08-24T01:05:29.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:05:29.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:05:29.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46909 2020-08-24T01:05:29.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:05:29.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:05:29.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: code=0 2020-08-24T01:05:29.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: teardown td/ec-error-rollforward 0 2020-08-24T01:05:29.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-08-24T01:05:29.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-08-24T01:05:29.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-08-24T01:05:29.597 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:05:29.597 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:05:29.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:05:29.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:05:29.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:05:29.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:05:29.598 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:05:29.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:05:29.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:05:29.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:05:29.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:05:29.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:05:29.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:05:29.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:05:29.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:05:29.600 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:05:29.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:05:29.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-08-24T01:05:29.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-08-24T01:05:29.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:05:29.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.46909 2020-08-24T01:05:29.601 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.46909 2020-08-24T01:05:29.601 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:05:29.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:05:29.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: return 0 2020-08-24T01:05:29.602 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-08-24T01:05:29.603 INFO:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-08-24T01:05:29.619 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-08-24T01:05:29.620 INFO:teuthology.orchestra.run.smithi060:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a941e55043ccee6d35a8717bec55ff7581e67d8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-08-24T01:05:29.681 INFO:tasks.workunit.client.0.smithi060.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-08-24T01:05:29.696 INFO:tasks.workunit.client.0.smithi060.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-08-24T01:05:29.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T01:05:29.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T01:05:29.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PYTHONWARNINGS=ignore 2020-08-24T01:05:29.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PYTHONWARNINGS=ignore 2020-08-24T01:05:29.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export CEPH_CONF=/dev/null 2020-08-24T01:05:29.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: CEPH_CONF=/dev/null 2020-08-24T01:05:29.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: unset CEPH_ARGS 2020-08-24T01:05:29.698 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: local code 2020-08-24T01:05:29.698 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: run td/osd-backfill-prio 2020-08-24T01:05:29.698 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-08-24T01:05:29.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-08-24T01:05:29.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-08-24T01:05:29.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-08-24T01:05:29.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-08-24T01:05:29.700 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-08-24T01:05:29.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none ' 2020-08-24T01:05:29.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:05:29.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:05:29.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-08-24T01:05:29.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-08-24T01:05:29.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-08-24T01:05:29.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-08-24T01:05:29.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-08-24T01:05:29.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-08-24T01:05:29.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-08-24T01:05:29.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-08-24T01:05:29.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-08-24T01:05:29.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-08-24T01:05:29.705 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-08-24T01:05:29.706 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-08-24T01:05:29.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-08-24T01:05:29.708 INFO:tasks.workunit.client.0.smithi060.stderr:TEST_backfill_priority' 2020-08-24T01:05:29.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-08-24T01:05:29.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-08-24T01:05:29.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-08-24T01:05:29.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-08-24T01:05:29.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-08-24T01:05:29.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:05:29.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-08-24T01:05:29.711 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:05:29.716 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:05:29.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:05:29.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:05:29.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:05:29.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:05:29.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:05:29.721 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:05:29.721 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:05:29.726 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:05:29.727 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:05:29.727 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:05:29.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:05:29.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:05:29.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:05:29.729 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:05:29.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:05:29.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:05:29.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-08-24T01:05:29.730 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:05:29.730 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.731 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:29.731 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50053 2020-08-24T01:05:29.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:05:29.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:05:29.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-08-24T01:05:29.741 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:05:29.742 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:29.742 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:29.742 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.50053 2020-08-24T01:05:29.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:05:29.743 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-08-24T01:05:29.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-08-24T01:05:29.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:05:29.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-08-24T01:05:29.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-08-24T01:05:29.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-08-24T01:05:29.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-08-24T01:05:29.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-08-24T01:05:29.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-08-24T01:05:29.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-08-24T01:05:29.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:05:29.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:05:29.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:05:29.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-08-24T01:05:29.751 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-08-24T01:05:30.031 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:05:30.031 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:30.031 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:30.032 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:30.032 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:30.032 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:30.032 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:05:30.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:05:30.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:05:30.210 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:05:30.211 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:05:30.211 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:05:30.211 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:05:30.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:05:30.212 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:05:30.212 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:05:30.212 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:05:30.213 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:05:30.213 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:30.214 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:30.214 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-mon.a.asok 2020-08-24T01:05:30.214 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:05:30.214 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.50053/ceph-mon.a.asok config get fsid 2020-08-24T01:05:30.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:05:30.315 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:05:30.315 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:05:30.315 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:05:30.317 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:05:30.317 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:05:30.317 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:05:30.318 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:05:30.318 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:05:30.318 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:30.318 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:30.318 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-mon.a.asok 2020-08-24T01:05:30.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:05:30.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.50053/ceph-mon.a.asok config get mon_host 2020-08-24T01:05:30.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-08-24T01:05:30.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-08-24T01:05:30.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:05:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:05:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:05:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-08-24T01:05:30.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:05:30.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:05:30.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:30.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:30.991 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:30.992 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:30.992 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:30.992 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:05:30.992 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:05:30.993 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:05:31.025 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-08-24T01:05:31.026 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-08-24T01:05:31.028 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-08-24T01:05:31.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:05:31.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-08-24T01:05:31.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-08-24T01:05:31.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:05:31.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:05:31.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:05:31.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-08-24T01:05:31.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:05:31.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:05:31.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:05:31.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:05:31.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-08-24T01:05:31.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-08-24T01:05:31.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:05:31.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:05:31.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-08-24T01:05:31.037 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:05:31.037 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:31.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:31.040 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:31.040 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:31.041 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:31.041 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:05:31.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:05:31.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:05:31.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:05:31.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:05:31.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-08-24T01:05:31.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-08-24T01:05:31.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:05:31.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:05:31.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:31.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:05:31.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:05:31.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-08-24T01:05:31.044 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:05:31.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=885f999e-4e9e-416c-9fb4-95d8ac393b7e 2020-08-24T01:05:31.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 885f999e-4e9e-416c-9fb4-95d8ac393b7e' 2020-08-24T01:05:31.045 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:05:31.045 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 885f999e-4e9e-416c-9fb4-95d8ac393b7e 2020-08-24T01:05:31.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBbEkNfiMN4AxAAcjPue2+nsyYKj2jzCHdAYA== 2020-08-24T01:05:31.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBbEkNfiMN4AxAAcjPue2+nsyYKj2jzCHdAYA=="}' 2020-08-24T01:05:31.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 885f999e-4e9e-416c-9fb4-95d8ac393b7e -i td/osd-backfill-prio/0/new.json 2020-08-24T01:05:31.386 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:31.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-08-24T01:05:31.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBbEkNfiMN4AxAAcjPue2+nsyYKj2jzCHdAYA== --osd-uuid 885f999e-4e9e-416c-9fb4-95d8ac393b7e 2020-08-24T01:05:31.432 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.427+0000 7f579cf39d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:31.434 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.427+0000 7f579cf39d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:31.436 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.431+0000 7f579cf39d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:31.437 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.431+0000 7f579cf39d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-08-24T01:05:31.437 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.431+0000 7f579cf39d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-08-24T01:05:31.438 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.431+0000 7f579cf39d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-08-24T01:05:31.475 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.467+0000 7f579cf39d80 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-08-24T01:05:31.873 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:31.867+0000 7f579cf39d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:05:34.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-08-24T01:05:34.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:05:34.556 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:05:34.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:05:34.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:05:34.806 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:05:34.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:05:34.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:34.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:05:34.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:05:34.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:05:34.836 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:34.831+0000 7f700022ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:34.839 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:34.831+0000 7f700022ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:34.840 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:34.835+0000 7f700022ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:35.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:05:35.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:35.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:05:35.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:35.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:35.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:35.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:35.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:35.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:05:35.163 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:35.373 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:35.367+0000 7f700022ed80 -1 Falling back to public interface 2020-08-24T01:05:35.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:05:36.145 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:36.139+0000 7f700022ed80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:05:36.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:05:36.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:36.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:05:36.538 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:05:36.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:36.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:05:36.917 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/50478,v1:127.0.0.1:6803/50478] [v2:127.0.0.1:6804/50478,v1:127.0.0.1:6805/50478] exists,up 885f999e-4e9e-416c-9fb4-95d8ac393b7e 2020-08-24T01:05:36.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:36.939 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:36.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:36.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:05:36.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-08-24T01:05:36.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-08-24T01:05:36.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:05:36.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:05:36.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:05:36.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-08-24T01:05:36.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:05:36.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:05:36.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:05:36.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:05:36.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-08-24T01:05:36.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-08-24T01:05:36.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:05:36.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:05:36.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-08-24T01:05:36.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:05:36.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:05:36.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:05:36.947 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:05:36.948 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:05:36.948 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:05:36.948 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:05:36.948 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:05:36.948 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:05:36.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:05:36.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:05:36.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-08-24T01:05:36.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-08-24T01:05:36.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:05:36.951 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:05:36.951 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:36.952 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:05:36.952 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:05:36.952 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-08-24T01:05:36.952 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:05:36.954 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=c2b859fb-2616-478c-8c1f-c3bfeb3b57ee 2020-08-24T01:05:36.954 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 c2b859fb-2616-478c-8c1f-c3bfeb3b57ee' 2020-08-24T01:05:36.954 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:05:36.954 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBgEkNfafK+NxAAonISilkjY7eZKGhZ7/azwA== 2020-08-24T01:05:36.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBgEkNfafK+NxAAonISilkjY7eZKGhZ7/azwA=="}' 2020-08-24T01:05:36.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new c2b859fb-2616-478c-8c1f-c3bfeb3b57ee -i td/osd-backfill-prio/1/new.json 2020-08-24T01:05:36.955 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 c2b859fb-2616-478c-8c1f-c3bfeb3b57ee 2020-08-24T01:05:37.410 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:05:37.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-08-24T01:05:37.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBgEkNfafK+NxAAonISilkjY7eZKGhZ7/azwA== --osd-uuid c2b859fb-2616-478c-8c1f-c3bfeb3b57ee 2020-08-24T01:05:37.455 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.447+0000 7fdcf85c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:37.458 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.451+0000 7fdcf85c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:37.459 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.451+0000 7fdcf85c0d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:37.459 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.455+0000 7fdcf85c0d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-08-24T01:05:37.460 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.455+0000 7fdcf85c0d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-08-24T01:05:37.460 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.455+0000 7fdcf85c0d80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-08-24T01:05:37.511 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.499+0000 7fdcf85c0d80 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-08-24T01:05:37.983 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:37.979+0000 7fdcf85c0d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:05:40.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-08-24T01:05:40.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:05:40.846 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:05:40.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:05:40.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:05:41.319 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:05:41.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:05:41.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:05:41.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:05:41.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:05:41.321 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:05:41.350 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:41.343+0000 7f25e9341d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:41.352 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:41.347+0000 7f25e9341d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:41.353 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:41.347+0000 7f25e9341d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:05:41.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:05:41.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:05:41.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:05:41.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:05:41.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:05:41.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:41.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:05:41.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:41.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:05:41.687 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:05:41.893 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:41.887+0000 7f25e9341d80 -1 Falling back to public interface 2020-08-24T01:05:42.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:05:42.719 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:05:42.711+0000 7f25e9341d80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:05:43.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:05:43.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:05:43.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:05:43.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:05:43.066 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:05:43.066 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:05:43.514 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/50833,v1:127.0.0.1:6811/50833] [v2:127.0.0.1:6812/50833,v1:127.0.0.1:6813/50833] exists,up c2b859fb-2616-478c-8c1f-c3bfeb3b57ee 2020-08-24T01:05:43.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:05:43.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:05:43.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:05:43.516 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-08-24T01:05:43.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:05:43.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-08-24T01:05:43.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-08-24T01:05:43.987 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test1' already exists 2020-08-24T01:05:43.997 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:05:45.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-08-24T01:05:52.893 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 size to 2 2020-08-24T01:05:52.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:05:52.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-08-24T01:05:52.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-08-24T01:05:52.895 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test2' already exists 2020-08-24T01:05:52.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:05:52.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-08-24T01:05:52.895 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 2 size to 2 2020-08-24T01:05:52.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:05:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-08-24T01:05:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-08-24T01:05:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test3' already exists 2020-08-24T01:05:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:05:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-08-24T01:05:52.896 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 3 size to 2 2020-08-24T01:05:52.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-08-24T01:05:55.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-08-24T01:05:55.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:05:55.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:05:55.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:05:55.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:05:55.176 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:05:55.177 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:05:55.177 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:05:55.177 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:05:55.177 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:05:55.178 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:05:55.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:05:55.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:05:55.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:05:55.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:05:55.381 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:05:55.752 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:05:55.753 INFO:tasks.workunit.client.0.smithi060.stderr:1' 2020-08-24T01:05:55.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:05:55.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:05:55.754 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:05:55.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-08-24T01:05:55.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836486 2020-08-24T01:05:55.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486' 2020-08-24T01:05:55.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:05:55.935 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:05:56.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-08-24T01:05:56.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705669 2020-08-24T01:05:56.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836486 1-38654705669' 2020-08-24T01:05:56.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:05:56.087 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836486 2020-08-24T01:05:56.088 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:05:56.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:05:56.091 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836486 2020-08-24T01:05:56.091 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:05:56.097 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836486 2020-08-24T01:05:56.097 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-08-24T01:05:56.097 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:05:56.098 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836486 2020-08-24T01:05:56.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836484 -lt 21474836486 2020-08-24T01:05:56.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:05:57.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:05:57.463 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:05:57.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836486 -lt 21474836486 2020-08-24T01:05:57.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:05:57.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:05:57.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705669 2020-08-24T01:05:57.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:05:57.851 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705669 2020-08-24T01:05:57.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:05:57.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705669 2020-08-24T01:05:57.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-08-24T01:05:57.855 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:05:57.855 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705669 2020-08-24T01:05:58.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705669 -lt 38654705669 2020-08-24T01:05:58.234 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:05:58.235 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:05:58.235 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:05:58.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-08-24T01:05:58.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:05:58.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:05:58.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:05:58.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:05:58.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:05:58.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:05:58.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:05:59.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-08-24T01:05:59.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:05:59.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:05:59.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:05:59.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-08-24T01:05:59.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:05:59.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:05:59.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-08-24T01:05:59.822 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:05:59.823 INFO:tasks.workunit.client.0.smithi060.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:05:50.329667+0000 0'0 24:16 [1,0] 1 [1,0] 1 0'0 2020-08-24T01:05:48.287007+0000 0'0 2020-08-24T01:05:48.287007+0000 0 2020-08-24T01:05:59.823 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:05:46.330922+0000 0'0 23:24 [1,0] 1 [1,0] 1 0'0 2020-08-24T01:05:43.811164+0000 0'0 2020-08-24T01:05:43.811164+0000 0 2020-08-24T01:05:59.823 INFO:tasks.workunit.client.0.smithi060.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:05:48.087680+0000 0'0 23:22 [0,1] 0 [0,1] 0 0'0 2020-08-24T01:05:46.062522+0000 0'0 2020-08-24T01:05:46.062522+0000 0 2020-08-24T01:05:59.823 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:05:59.824 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:05:59.825 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:05:59.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-08-24T01:05:59.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-08-24T01:05:59.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-08-24T01:05:59.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-08-24T01:05:59.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-08-24T01:05:59.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-08-24T01:05:59.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-08-24T01:05:59.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-08-24T01:05:59.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-08-24T01:05:59.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-08-24T01:05:59.836 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-08-24T01:05:59.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:05:59.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-08-24T01:05:59.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-08-24T01:06:00.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:06:00.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-08-24T01:06:00.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:06:00.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-08-24T01:06:00.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-08-24T01:06:00.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-08-24T01:06:00.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-08-24T01:06:00.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-08-24T01:06:00.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-08-24T01:06:00.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-08-24T01:06:00.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:06:00.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-08-24T01:06:00.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-08-24T01:06:00.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:06:00.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-08-24T01:06:00.591 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:06:00.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-08-24T01:06:00.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-08-24T01:06:00.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-08-24T01:06:00.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-08-24T01:06:00.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-08-24T01:06:00.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-08-24T01:06:00.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-08-24T01:06:00.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-08-24T01:06:00.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-08-24T01:06:00.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-08-24T01:06:00.601 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-08-24T01:06:00.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-08-24T01:06:00.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:06:00.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-08-24T01:06:00.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:06:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-08-24T01:06:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-08-24T01:06:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-08-24T01:06:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-08-24T01:06:00.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-08-24T01:06:00.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-08-24T01:06:01.098 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test3' does not exist 2020-08-24T01:06:01.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-08-24T01:06:01.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-08-24T01:06:01.110 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-08-24T01:06:01.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-08-24T01:06:01.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-08-24T01:06:01.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-08-24T01:06:01.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-08-24T01:06:01.887 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 size to 1 2020-08-24T01:06:01.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-08-24T01:06:02.773 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 recovery_priority to 1 2020-08-24T01:06:02.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-08-24T01:06:03.461 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 2 size to 1 2020-08-24T01:06:03.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-08-24T01:06:04.187 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 2 recovery_priority to 2 2020-08-24T01:06:04.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-08-24T01:06:04.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:06:04.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:06:04.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:06:04.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:06:04.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:06:04.206 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:06:04.206 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:06:04.206 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:06:04.206 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:06:04.206 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:06:04.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:06:04.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:06:04.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:06:04.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:06:04.397 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:06:04.761 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:06:04.762 INFO:tasks.workunit.client.0.smithi060.stderr:1' 2020-08-24T01:06:04.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:06:04.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:06:04.763 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:06:04.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-08-24T01:06:04.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836490 2020-08-24T01:06:04.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836490' 2020-08-24T01:06:04.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:06:04.924 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:06:05.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705673 2020-08-24T01:06:05.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705673 2020-08-24T01:06:05.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836490 1-38654705673' 2020-08-24T01:06:05.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:06:05.085 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836490 2020-08-24T01:06:05.086 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:06:05.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:06:05.089 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836490 2020-08-24T01:06:05.089 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:06:05.098 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836490 2020-08-24T01:06:05.098 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-08-24T01:06:05.098 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:06:05.099 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836490 2020-08-24T01:06:05.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836488 -lt 21474836490 2020-08-24T01:06:05.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:06:06.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:06:06.485 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:06:06.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836490 -lt 21474836490 2020-08-24T01:06:06.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:06:06.866 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705673 2020-08-24T01:06:06.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:06:06.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:06:06.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705673 2020-08-24T01:06:06.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:06:06.872 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705673 2020-08-24T01:06:06.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705673 2020-08-24T01:06:06.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2020-08-24T01:06:06.875 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:06:07.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705673 -lt 38654705673 2020-08-24T01:06:07.258 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:06:07.258 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:06:07.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:06:07.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-08-24T01:06:11.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:06:11.301 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:06:11.302 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:06:11.302 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:06:11.302 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:06:11.302 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:06:11.302 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:06:11.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:06:11.303 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:06:11.303 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:06:11.303 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:06:11.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-08-24T01:06:11.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:06:11.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:06:11.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-08-24T01:06:11.304 INFO:tasks.workunit.client.0.smithi060.stderr:10+0 records in 2020-08-24T01:06:11.304 INFO:tasks.workunit.client.0.smithi060.stderr:10+0 records out 2020-08-24T01:06:11.305 INFO:tasks.workunit.client.0.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0542612 s, 193 MB/s 2020-08-24T01:06:11.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-08-24T01:06:11.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-08-24T01:06:11.305 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-08-24T01:06:11.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.340 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-08-24T01:06:11.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:11.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-08-24T01:06:12.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:12.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-08-24T01:06:12.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:12.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-08-24T01:06:12.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:12.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-08-24T01:06:13.230 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:13.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-08-24T01:06:13.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:13.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-08-24T01:06:13.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:13.964 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-08-24T01:06:14.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:14.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-08-24T01:06:14.751 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:14.752 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-08-24T01:06:15.096 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:15.096 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-08-24T01:06:15.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:15.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-08-24T01:06:15.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:15.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-08-24T01:06:16.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:16.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-08-24T01:06:16.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:16.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-08-24T01:06:16.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:16.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-08-24T01:06:17.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:17.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-08-24T01:06:17.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:17.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-08-24T01:06:18.039 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:18.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-08-24T01:06:18.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:18.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-08-24T01:06:18.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:18.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-08-24T01:06:19.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:19.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-08-24T01:06:19.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:19.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-08-24T01:06:19.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:19.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-08-24T01:06:20.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:20.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-08-24T01:06:20.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:20.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-08-24T01:06:21.067 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:21.067 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-08-24T01:06:21.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:21.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-08-24T01:06:21.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:21.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-08-24T01:06:22.153 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:22.153 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-08-24T01:06:22.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:22.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-08-24T01:06:22.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:22.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-08-24T01:06:23.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:23.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-08-24T01:06:23.692 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:23.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-08-24T01:06:24.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:24.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-08-24T01:06:24.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:24.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-08-24T01:06:24.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:24.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-08-24T01:06:25.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:25.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-08-24T01:06:25.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:25.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-08-24T01:06:25.982 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:25.983 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-08-24T01:06:26.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:26.321 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-08-24T01:06:26.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:26.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-08-24T01:06:27.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:27.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-08-24T01:06:27.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-08-24T01:06:27.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-08-24T01:06:27.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-08-24T01:06:27.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-08-24T01:06:27.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:27.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-08-24T01:06:27.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:27.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-08-24T01:06:28.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:28.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-08-24T01:06:28.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:28.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-08-24T01:06:28.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:28.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-08-24T01:06:29.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:29.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-08-24T01:06:29.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:29.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-08-24T01:06:30.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:30.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-08-24T01:06:30.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:30.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-08-24T01:06:30.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:30.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-08-24T01:06:31.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:31.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-08-24T01:06:31.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:31.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-08-24T01:06:31.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:31.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-08-24T01:06:32.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:32.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-08-24T01:06:32.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:32.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-08-24T01:06:33.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:33.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-08-24T01:06:33.772 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:33.772 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-08-24T01:06:34.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:34.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-08-24T01:06:34.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:34.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-08-24T01:06:34.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:34.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-08-24T01:06:35.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:35.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-08-24T01:06:35.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:35.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-08-24T01:06:36.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:36.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-08-24T01:06:36.365 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:36.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-08-24T01:06:36.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:36.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-08-24T01:06:37.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:37.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-08-24T01:06:37.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:37.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-08-24T01:06:37.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:37.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-08-24T01:06:38.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:38.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-08-24T01:06:38.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:38.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-08-24T01:06:39.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:39.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-08-24T01:06:39.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:39.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-08-24T01:06:39.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:39.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-08-24T01:06:40.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:40.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-08-24T01:06:40.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:40.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-08-24T01:06:40.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:40.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-08-24T01:06:41.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:41.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-08-24T01:06:41.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.396 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-08-24T01:06:53.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-08-24T01:06:53.411 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-08-24T01:06:53.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-08-24T01:06:53.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-08-24T01:06:53.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-08-24T01:06:53.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-08-24T01:06:53.413 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-08-24T01:06:53.413 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-08-24T01:06:53.413 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-08-24T01:06:53.413 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-08-24T01:06:53.414 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-08-24T01:06:53.414 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-08-24T01:06:53.414 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-08-24T01:06:53.414 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-08-24T01:06:53.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-08-24T01:06:53.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-08-24T01:06:53.415 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:06:53.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-08-24T01:06:53.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-08-24T01:06:53.416 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 size to 2 2020-08-24T01:06:53.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-08-24T01:06:53.416 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 2 size to 2 2020-08-24T01:06:53.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-08-24T01:06:53.417 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:06:53.417 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-08-24T01:06:02.189358+0000 34'150 34:191 [1] 1 [1] 1 0'0 2020-08-24T01:05:43.811164+0000 0'0 2020-08-24T01:05:43.811164+0000 0 2020-08-24T01:06:53.418 INFO:tasks.workunit.client.0.smithi060.stdout:2.0 39 0 0 0 0 408944640 0 0 17 17 active+clean 2020-08-24T01:06:03.595183+0000 34'117 34:153 [0] 0 [0] 0 0'0 2020-08-24T01:05:46.062522+0000 0'0 2020-08-24T01:05:46.062522+0000 0 2020-08-24T01:06:53.418 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:06:53.419 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:06:54.323 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-08-24T01:06:54.324 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:06:54.324 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:06:54.325 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:06:54.325 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:06:54.326 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:06:54.326 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:06:54.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-08-24T01:06:54.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:06:54.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-08-24T01:06:54.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-08-24T01:06:54.436 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 2020-08-24T01:06:54.437 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:06:54.437 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:06:54.438 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:06:54.438 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:06:54.438 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:06:54.438 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:06:54.438 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:06:54.439 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:06:54.439 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 152, 2020-08-24T01:06:54.439 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:06:54.439 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:06:54.439 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:06:54.439 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:06:54.440 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:06:54.440 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:06:54.440 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:06:54.440 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:06:54.440 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:06:54.441 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:06:54.441 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "2.0", 2020-08-24T01:06:54.441 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 153, 2020-08-24T01:06:54.441 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:06:54.441 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:06:54.442 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:06:54.442 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:06:54.442 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:06:54.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-08-24T01:06:54.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-08-24T01:06:54.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-08-24T01:06:54.444 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:06:54.444 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:06:54.444 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:06:54.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.0.asok 2020-08-24T01:06:54.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-08-24T01:06:54.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.0.asok dump_recovery_reservations 2020-08-24T01:06:54.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-08-24T01:06:54.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-08-24T01:06:54.548 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 2020-08-24T01:06:54.548 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:06:54.548 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:06:54.549 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:06:54.549 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:06:54.549 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:06:54.549 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:06:54.549 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:06:54.550 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "2.0", 2020-08-24T01:06:54.550 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 153, 2020-08-24T01:06:54.550 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:06:54.550 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:06:54.550 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:06:54.551 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:06:54.551 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:06:54.551 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:06:54.551 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:06:54.551 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:06:54.551 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:06:54.552 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:06:54.552 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:06:54.552 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 152, 2020-08-24T01:06:54.552 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:06:54.552 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:06:54.553 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:06:54.553 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:06:54.553 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:06:54.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-08-24T01:06:54.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-08-24T01:06:54.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-08-24T01:06:54.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-08-24T01:06:54.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-08-24T01:06:54.556 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-08-24T01:06:54.557 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-08-24T01:06:54.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-08-24T01:06:54.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-08-24T01:06:54.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-08-24T01:06:54.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-08-24T01:06:54.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-08-24T01:06:54.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-08-24T01:06:54.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-08-24T01:06:54.573 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-08-24T01:06:54.574 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-08-24T01:06:54.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-08-24T01:06:54.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-08-24T01:06:54.579 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-08-24T01:06:54.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-08-24T01:06:54.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-08-24T01:06:54.586 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-08-24T01:06:54.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-08-24T01:06:54.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-08-24T01:06:54.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-08-24T01:06:54.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-08-24T01:06:54.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-08-24T01:06:54.594 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-08-24T01:06:54.594 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-08-24T01:06:54.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-08-24T01:06:54.597 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-08-24T01:06:54.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-08-24T01:06:54.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-08-24T01:06:54.600 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-08-24T01:06:54.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-08-24T01:06:54.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-08-24T01:06:54.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-08-24T01:06:54.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:06:54.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:06:54.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:06:54.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:06:54.608 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:06:54.609 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:06:54.609 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:06:54.609 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:06:54.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:06:54.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:06:54.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:06:54.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:06:54.820 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:06:54.820 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:06:54.820 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:06:55.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:06:55.189 INFO:tasks.workunit.client.0.smithi060.stderr:1' 2020-08-24T01:06:55.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:06:55.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:06:55.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:06:55.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836502 2020-08-24T01:06:55.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836502 2020-08-24T01:06:55.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836502' 2020-08-24T01:06:55.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:06:55.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:06:55.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705685 2020-08-24T01:06:55.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705685 2020-08-24T01:06:55.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836502 1-38654705685' 2020-08-24T01:06:55.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:06:55.631 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836502 2020-08-24T01:06:55.632 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:06:55.634 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:06:55.636 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836502 2020-08-24T01:06:55.638 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:06:55.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836502 2020-08-24T01:06:55.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836502' 2020-08-24T01:06:55.641 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:06:55.642 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836502 2020-08-24T01:06:56.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836500 -lt 21474836502 2020-08-24T01:06:56.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:06:57.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:06:57.144 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:06:57.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836502 -lt 21474836502 2020-08-24T01:06:57.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:06:57.592 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705685 2020-08-24T01:06:57.592 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:06:57.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:06:57.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705685 2020-08-24T01:06:57.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:06:57.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705685 2020-08-24T01:06:57.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705685' 2020-08-24T01:06:57.598 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705685 2020-08-24T01:06:57.600 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:06:58.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705685 -lt 38654705685 2020-08-24T01:06:58.164 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:06:58.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:06:58.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:06:58.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-08-24T01:06:58.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:06:58.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:06:58.651 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:06:58.652 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:06:58.652 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:06:58.652 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:06:58.653 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:06:59.006 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:06:59.008 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:06:59.010 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:06:59.011 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:06:59.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:06:59.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T01:06:59.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:06:59.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T01:06:59.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:06:59.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:06:59.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:06:59.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:06:59.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:06:59.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:06:59.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:06:59.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:06:59.603 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:06:59.603 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:06:59.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:06:59.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:06:59.948 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:06:59.948 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:00.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:00.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:00.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:00.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:00.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:00.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:00.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:00.467 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:00.467 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:00.954 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22452253 2020-08-24T01:07:00.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22452253 '!=' null 2020-08-24T01:07:00.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:00.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:00.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:01.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:01.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:01.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:01.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:01.062 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:01.062 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:01.063 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:01.063 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:01.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:01.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:01.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:01.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:01.930 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:01.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:01.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:01.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:01.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:01.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:01.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:01.941 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:01.941 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:02.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22452253 2020-08-24T01:07:02.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22452253 '!=' null 2020-08-24T01:07:02.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:02.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:02.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:02.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:02.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:02.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:02.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:02.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:02.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:02.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:02.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:02.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:02.866 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:02.866 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:02.867 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:03.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:03.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:03.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:03.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:03.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:03.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:03.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:03.328 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:03.328 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:03.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26207319 2020-08-24T01:07:03.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26207319 '!=' null 2020-08-24T01:07:03.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:03.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:03.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:03.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:03.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:03.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:03.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:03.900 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:03.900 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:03.900 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:03.900 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:04.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:04.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:04.435 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:04.436 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:04.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:04.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:04.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:04.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:04.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:04.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:04.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:04.989 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:04.989 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:05.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29701959 2020-08-24T01:07:05.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29701959 '!=' null 2020-08-24T01:07:05.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:05.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:05.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:05.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:05.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:05.577 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:05.577 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:05.578 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:05.578 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:05.578 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:05.579 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:05.982 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:05.983 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:05.983 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:05.983 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:06.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:06.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:06.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:06.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:06.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:06.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:06.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:06.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:06.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:06.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24460656 2020-08-24T01:07:06.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24460656 '!=' null 2020-08-24T01:07:06.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:06.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:06.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:06.986 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:06.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:06.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:06.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:06.990 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:06.991 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:06.991 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:06.991 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:07.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:07.372 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:07.372 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:07.372 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:07.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:07.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:07.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:07.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:07.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:07.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:07.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:07.909 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:07.910 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:08.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24460656 2020-08-24T01:07:08.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24460656 '!=' null 2020-08-24T01:07:08.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:08.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:08.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:08.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:08.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:08.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:08.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:08.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:08.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:08.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:08.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:08.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:08.825 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:08.826 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:08.826 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:09.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:09.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:09.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:09.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:09.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:09.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:09.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:09.383 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:09.383 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:09.857 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24461405 2020-08-24T01:07:09.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24461405 '!=' null 2020-08-24T01:07:09.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:09.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:09.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:09.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:09.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:09.965 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:09.966 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:09.966 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:09.967 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:09.967 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:09.967 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:10.364 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:10.365 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:10.365 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:10.366 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:10.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:10.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:10.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:10.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:10.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:10.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:10.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:10.823 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:10.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:11.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27956046 2020-08-24T01:07:11.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27956046 '!=' null 2020-08-24T01:07:11.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:11.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:11.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:11.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:11.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:11.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:11.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:11.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:11.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:11.455 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:11.455 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:11.832 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:11.833 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:11.833 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:11.833 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:12.333 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:12.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:12.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:12.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:12.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:12.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:12.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:12.336 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:12.336 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:12.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29703169 2020-08-24T01:07:12.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29703169 '!=' null 2020-08-24T01:07:12.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:12.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:12.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:12.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:12.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:12.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:12.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:12.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:12.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:12.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:12.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:13.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:14.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:14.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:14.611 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:14.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:14.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:14.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:14.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:14.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:14.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:14.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:14.612 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:14.612 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:14.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29703169 2020-08-24T01:07:14.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29703169 '!=' null 2020-08-24T01:07:14.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:14.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:14.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:14.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:14.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:14.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:14.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:14.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:14.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:14.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:14.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:14.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:14.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:14.744 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:14.744 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:15.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:15.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:15.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:15.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:15.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:15.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:15.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:15.209 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:15.210 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:15.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28829584 2020-08-24T01:07:15.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28829584 '!=' null 2020-08-24T01:07:15.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:15.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:15.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:15.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:15.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:15.821 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:15.822 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:15.822 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:15.822 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:15.822 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:15.823 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:16.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:16.295 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:16.295 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:16.296 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:16.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:16.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:16.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:16.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:16.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:16.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:16.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:16.891 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:16.892 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:17.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22713716 2020-08-24T01:07:17.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22713716 '!=' null 2020-08-24T01:07:17.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:17.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:17.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:17.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:17.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:17.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:17.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:17.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:17.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:17.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:17.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:17.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:17.813 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:17.813 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:17.814 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:18.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:18.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:18.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:18.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:18.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:18.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:18.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:18.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:18.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:18.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28828289 2020-08-24T01:07:18.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28828289 '!=' null 2020-08-24T01:07:18.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:18.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:18.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:19.025 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:19.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:19.029 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:19.029 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:19.029 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:19.030 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:19.030 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:19.031 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:19.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:19.398 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:19.398 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:19.399 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:19.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:19.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:19.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:19.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:19.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:19.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:19.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:19.858 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:19.858 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:20.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28828289 2020-08-24T01:07:20.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28828289 '!=' null 2020-08-24T01:07:20.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:20.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:20.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:20.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:20.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:20.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:20.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:20.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:20.408 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:20.408 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:20.408 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:20.761 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:20.763 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:20.764 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:20.764 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:21.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:21.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:21.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:21.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:21.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:21.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:21.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:21.355 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:21.355 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:21.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29701175 2020-08-24T01:07:21.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29701175 '!=' null 2020-08-24T01:07:21.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:21.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:21.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:21.919 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:21.919 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:21.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:21.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:21.923 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:21.923 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:21.924 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:21.924 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:22.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:22.272 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:22.272 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:22.273 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:22.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:22.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:22.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:22.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:22.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:22.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:22.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:22.821 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:22.821 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:23.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29701457 2020-08-24T01:07:23.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29701457 '!=' null 2020-08-24T01:07:23.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:23.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:23.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:23.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:23.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:23.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:23.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:23.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:23.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:23.423 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:23.423 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:23.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:23.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:23.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:23.785 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:24.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:24.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:24.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:24.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:24.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:24.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:24.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:24.242 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:24.242 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:24.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29701457 2020-08-24T01:07:24.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29701457 '!=' null 2020-08-24T01:07:24.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:24.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:24.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:24.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:24.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:24.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:24.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:24.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:24.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:24.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:24.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:25.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:25.230 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:25.231 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:25.232 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:25.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:25.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:25.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:25.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:25.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:25.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:25.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:25.736 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:25.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:26.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29701404 2020-08-24T01:07:26.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29701404 '!=' null 2020-08-24T01:07:26.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:26.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:26.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:26.348 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:26.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:26.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:26.351 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:26.351 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:26.351 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:26.352 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:26.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:26.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:26.711 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:26.712 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:26.712 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:27.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:27.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:27.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:27.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:27.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:27.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:27.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:27.255 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:27.256 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:27.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29351879 2020-08-24T01:07:27.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29351879 '!=' null 2020-08-24T01:07:27.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:27.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:27.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:27.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:27.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:27.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:27.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:27.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:27.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:27.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:27.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:28.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:28.232 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:28.232 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:28.233 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:28.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:28.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:28.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:28.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:28.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:28.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:28.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:28.681 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:28.681 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:29.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15724404 2020-08-24T01:07:29.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15724404 '!=' null 2020-08-24T01:07:29.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:29.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:29.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:29.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:29.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:29.226 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:29.227 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:29.227 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:29.227 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:29.227 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:29.227 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:29.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:07:29.575 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:29.575 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:29.575 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:30.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-08-24T01:07:30.024 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:07:30.024 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:30.024 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:30.024 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:30.025 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:30.025 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:30.025 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:30.026 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:30.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15724404 2020-08-24T01:07:30.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15724404 '!=' null 2020-08-24T01:07:30.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:30.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:30.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:30.567 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:30.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:30.569 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:30.569 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:30.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:30.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:30.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:30.571 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:30.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:07:30.941 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:30.941 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:30.942 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:31.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-08-24T01:07:31.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-08-24T01:07:31.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:07:31.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-08-24T01:07:31.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:31.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:31.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:31.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:31.624 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:31.624 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:31.625 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:31.626 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:31.627 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:31.627 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:31.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:07:31.987 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:31.988 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:31.988 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:32.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-08-24T01:07:32.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:07:32.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:07:32.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:07:32.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:07:32.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:07:32.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:07:32.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:07:32.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:07:32.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=15724477 2020-08-24T01:07:32.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 15724477 '!=' null 2020-08-24T01:07:32.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:07:32.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:07:32.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:07:32.983 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:07:32.984 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:07:32.986 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:07:32.986 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:07:32.986 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:07:32.986 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:07:32.987 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:07:32.987 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:07:33.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:07:33.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:07:33.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:07:33.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:07:33.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-08-24T01:07:33.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:07:33.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:07:33.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-08-24T01:07:33.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-08-24T01:07:33.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-08-24T01:07:33.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-08-24T01:07:33.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-08-24T01:07:33.795 INFO:tasks.workunit.client.0.smithi060.stdout:TEST PASSED 2020-08-24T01:07:34.261 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test1' does not exist 2020-08-24T01:07:34.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-08-24T01:07:34.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-08-24T01:07:34.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-08-24T01:07:34.932 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test2' does not exist 2020-08-24T01:07:34.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-08-24T01:07:34.944 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:07:34.945 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:07:34.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:07:34.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:07:34.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:07:35.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:07:35.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-08-24T01:07:35.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-08-24T01:07:35.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-08-24T01:07:35.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:07:35.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-08-24T01:07:35.062 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:07:35.062 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:07:35.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:07:35.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:07:35.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:07:35.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:07:35.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:07:35.077 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:07:35.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:07:35.077 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:07:35.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:07:35.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:07:35.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:07:35.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:07:35.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:07:35.079 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:07:35.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:07:35.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:07:35.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-08-24T01:07:35.099 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:07:35.100 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:35.100 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:35.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50053 2020-08-24T01:07:35.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:07:35.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:07:35.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-08-24T01:07:35.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-08-24T01:07:35.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-08-24T01:07:35.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-08-24T01:07:35.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-08-24T01:07:35.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:07:35.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-08-24T01:07:35.109 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:07:35.109 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:07:35.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:07:35.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:07:35.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:07:35.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:07:35.110 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:07:35.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:07:35.111 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:07:35.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:07:35.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:07:35.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:07:35.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:07:35.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:07:35.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:07:35.113 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:07:35.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:07:35.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:07:35.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-08-24T01:07:35.118 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:07:35.119 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:35.119 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:35.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50053 2020-08-24T01:07:35.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:07:35.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:07:35.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-08-24T01:07:35.123 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:07:35.123 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:35.123 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:35.124 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.50053 2020-08-24T01:07:35.125 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:07:35.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-08-24T01:07:35.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:07:35.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-08-24T01:07:35.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-08-24T01:07:35.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-08-24T01:07:35.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-08-24T01:07:35.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-08-24T01:07:35.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-08-24T01:07:35.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-08-24T01:07:35.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-08-24T01:07:35.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-08-24T01:07:35.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-08-24T01:07:35.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:07:35.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:07:35.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:07:35.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-08-24T01:07:35.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-08-24T01:07:35.404 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:07:35.405 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:07:35.405 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:07:35.407 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:07:35.407 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:35.407 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:35.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:35.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:07:35.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:07:35.568 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:07:35.568 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:07:35.568 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:07:35.568 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:07:35.569 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:07:35.569 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:07:35.570 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:07:35.570 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:07:35.571 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:07:35.571 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:35.572 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:35.572 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-mon.a.asok 2020-08-24T01:07:35.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:07:35.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.50053/ceph-mon.a.asok config get fsid 2020-08-24T01:07:35.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:07:35.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:07:35.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:07:35.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:07:35.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:07:35.686 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:07:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:07:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:07:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:07:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:35.688 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-mon.a.asok 2020-08-24T01:07:35.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:07:35.688 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.50053/ceph-mon.a.asok config get mon_host 2020-08-24T01:07:35.772 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-08-24T01:07:35.772 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-08-24T01:07:35.773 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:07:35.773 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:07:35.773 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:07:35.773 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-08-24T01:07:35.773 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:07:36.267 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:07:36.268 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:07:36.268 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:07:36.269 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:07:36.269 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:36.269 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:36.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:36.270 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:07:36.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:07:36.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-08-24T01:07:36.298 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-08-24T01:07:36.301 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-08-24T01:07:36.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:07:36.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-08-24T01:07:36.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-08-24T01:07:36.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:07:36.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:07:36.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:07:36.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-08-24T01:07:36.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:07:36.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:07:36.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:07:36.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:07:36.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-08-24T01:07:36.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-08-24T01:07:36.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:07:36.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:07:36.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-08-24T01:07:36.306 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:07:36.306 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:07:36.307 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:07:36.309 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:07:36.309 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:36.310 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:36.310 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:36.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:36.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:07:36.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:07:36.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:07:36.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-08-24T01:07:36.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-08-24T01:07:36.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:07:36.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:07:36.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:07:36.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:07:36.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:07:36.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-08-24T01:07:36.312 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:07:36.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=380ba047-0743-4cc1-8021-1fd3c76d197b 2020-08-24T01:07:36.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 380ba047-0743-4cc1-8021-1fd3c76d197b' 2020-08-24T01:07:36.313 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:07:36.313 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 380ba047-0743-4cc1-8021-1fd3c76d197b 2020-08-24T01:07:36.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDYEkNf9BxqExAAdBnIIDPX5OwtRJlpPOrFgA== 2020-08-24T01:07:36.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDYEkNf9BxqExAAdBnIIDPX5OwtRJlpPOrFgA=="}' 2020-08-24T01:07:36.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 380ba047-0743-4cc1-8021-1fd3c76d197b -i td/osd-backfill-prio/0/new.json 2020-08-24T01:07:36.655 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:07:36.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-08-24T01:07:36.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDYEkNf9BxqExAAdBnIIDPX5OwtRJlpPOrFgA== --osd-uuid 380ba047-0743-4cc1-8021-1fd3c76d197b 2020-08-24T01:07:36.694 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:36.690+0000 7fe0b52a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:36.698 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:36.694+0000 7fe0b52a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:36.700 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:36.698+0000 7fe0b52a2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:36.701 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:36.698+0000 7fe0b52a2d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-08-24T01:07:36.701 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:36.698+0000 7fe0b52a2d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-08-24T01:07:36.701 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:36.698+0000 7fe0b52a2d80 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-08-24T01:07:36.743 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:36.738+0000 7fe0b52a2d80 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-08-24T01:07:37.120 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:37.118+0000 7fe0b52a2d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:07:39.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-08-24T01:07:39.812 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:07:39.813 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:07:39.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:07:39.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:07:40.100 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:07:40.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:07:40.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:07:40.101 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:07:40.101 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:07:40.101 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:07:40.134 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:40.130+0000 7fb0529e6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:40.136 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:40.134+0000 7fb0529e6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:40.138 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:40.134+0000 7fb0529e6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:40.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:07:40.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:07:40.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:07:40.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:07:40.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:07:40.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:07:40.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:07:40.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:07:40.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:07:40.454 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:07:40.669 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:40.666+0000 7fb0529e6d80 -1 Falling back to public interface 2020-08-24T01:07:40.830 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:07:41.469 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:41.466+0000 7fb0529e6d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:07:41.832 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:07:41.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:07:41.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:07:41.836 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:07:41.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:07:41.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:07:42.210 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/57342,v1:127.0.0.1:6803/57342] [v2:127.0.0.1:6804/57342,v1:127.0.0.1:6805/57342] exists,up 380ba047-0743-4cc1-8021-1fd3c76d197b 2020-08-24T01:07:42.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:07:42.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:07:42.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:07:42.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:07:42.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-08-24T01:07:42.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-08-24T01:07:42.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:07:42.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:07:42.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:07:42.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-08-24T01:07:42.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:07:42.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:07:42.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:07:42.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:07:42.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-08-24T01:07:42.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-08-24T01:07:42.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:07:42.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:07:42.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-08-24T01:07:42.215 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:07:42.215 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:07:42.215 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:07:42.218 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:07:42.218 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:42.219 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:42.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:42.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:42.220 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:07:42.220 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:07:42.220 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:07:42.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-08-24T01:07:42.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-08-24T01:07:42.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:07:42.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:07:42.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:07:42.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:07:42.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:07:42.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-08-24T01:07:42.223 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:07:42.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=f44b58ab-c4ec-4af7-b13e-3d4e681d7071 2020-08-24T01:07:42.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 f44b58ab-c4ec-4af7-b13e-3d4e681d7071' 2020-08-24T01:07:42.224 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:07:42.225 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 f44b58ab-c4ec-4af7-b13e-3d4e681d7071 2020-08-24T01:07:42.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDeEkNfEHD+DRAAvrabObW1uLOZlyi9nUriTQ== 2020-08-24T01:07:42.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDeEkNfEHD+DRAAvrabObW1uLOZlyi9nUriTQ=="}' 2020-08-24T01:07:42.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new f44b58ab-c4ec-4af7-b13e-3d4e681d7071 -i td/osd-backfill-prio/1/new.json 2020-08-24T01:07:42.685 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:07:42.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-08-24T01:07:42.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDeEkNfEHD+DRAAvrabObW1uLOZlyi9nUriTQ== --osd-uuid f44b58ab-c4ec-4af7-b13e-3d4e681d7071 2020-08-24T01:07:42.736 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:42.730+0000 7f07e458bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:42.739 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:42.734+0000 7f07e458bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:42.740 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:42.738+0000 7f07e458bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:42.741 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:42.738+0000 7f07e458bd80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-08-24T01:07:42.741 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:42.738+0000 7f07e458bd80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-08-24T01:07:42.741 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:42.738+0000 7f07e458bd80 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-08-24T01:07:42.779 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:42.774+0000 7f07e458bd80 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-08-24T01:07:43.239 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:43.238+0000 7f07e458bd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:07:46.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-08-24T01:07:47.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:07:47.493 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:07:47.493 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:07:47.494 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:07:47.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:07:47.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:07:47.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:07:47.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:07:47.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:07:47.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:07:47.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:07:47.495 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:46.778+0000 7f70a64c6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:47.495 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:46.782+0000 7f70a64c6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:47.496 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:46.782+0000 7f70a64c6d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:47.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:07:47.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:07:47.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:07:47.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:07:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:07:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:07:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:07:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:07:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:07:47.498 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:47.326+0000 7f70a64c6d80 -1 Falling back to public interface 2020-08-24T01:07:47.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:07:48.100 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:48.098+0000 7f70a64c6d80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:07:48.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:07:48.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:07:48.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:07:48.480 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:07:48.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:07:48.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:07:48.857 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/57697,v1:127.0.0.1:6811/57697] [v2:127.0.0.1:6812/57697,v1:127.0.0.1:6813/57697] exists,up f44b58ab-c4ec-4af7-b13e-3d4e681d7071 2020-08-24T01:07:48.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:07:48.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:07:48.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:07:48.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:07:48.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-08-24T01:07:48.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-08-24T01:07:48.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:07:48.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T01:07:48.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:07:48.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-08-24T01:07:48.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:07:48.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:07:48.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:07:48.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:07:48.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-08-24T01:07:48.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-08-24T01:07:48.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:07:48.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:07:48.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-08-24T01:07:48.862 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:07:48.865 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:07:48.865 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:07:48.865 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:07:48.866 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:48.866 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:48.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:48.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:48.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:07:48.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:07:48.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:07:48.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-08-24T01:07:48.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-08-24T01:07:48.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:07:48.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:07:48.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:07:48.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:07:48.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:07:48.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-08-24T01:07:48.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:07:48.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1f6ffa51-a8e7-48be-9aae-946bfa091bf1 2020-08-24T01:07:48.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 1f6ffa51-a8e7-48be-9aae-946bfa091bf1' 2020-08-24T01:07:48.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:07:48.872 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 1f6ffa51-a8e7-48be-9aae-946bfa091bf1 2020-08-24T01:07:48.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDkEkNf+F9yNBAAHw8zt/D388uG+rBKn7e8/Q== 2020-08-24T01:07:48.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDkEkNf+F9yNBAAHw8zt/D388uG+rBKn7e8/Q=="}' 2020-08-24T01:07:48.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1f6ffa51-a8e7-48be-9aae-946bfa091bf1 -i td/osd-backfill-prio/2/new.json 2020-08-24T01:07:49.468 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:07:49.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-08-24T01:07:49.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDkEkNf+F9yNBAAHw8zt/D388uG+rBKn7e8/Q== --osd-uuid 1f6ffa51-a8e7-48be-9aae-946bfa091bf1 2020-08-24T01:07:49.516 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:49.514+0000 7f302df0dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:49.519 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:49.514+0000 7f302df0dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:49.520 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:49.518+0000 7f302df0dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:49.521 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:49.518+0000 7f302df0dd80 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-08-24T01:07:49.521 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:49.518+0000 7f302df0dd80 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-08-24T01:07:49.521 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:49.518+0000 7f302df0dd80 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-08-24T01:07:49.595 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:49.590+0000 7f302df0dd80 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-08-24T01:07:50.042 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:50.038+0000 7f302df0dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:07:52.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-08-24T01:07:52.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:07:52.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T01:07:52.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:07:52.924 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T01:07:53.389 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:07:53.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T01:07:53.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:07:53.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:07:53.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:07:53.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:07:53.421 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:53.418+0000 7fb26ad21d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:53.423 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:53.422+0000 7fb26ad21d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:53.425 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:53.422+0000 7fb26ad21d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:53.747 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T01:07:53.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:07:53.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:07:53.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:07:53.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:07:53.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:07:53.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:07:53.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:07:53.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:07:53.750 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:07:53.968 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:53.966+0000 7fb26ad21d80 -1 Falling back to public interface 2020-08-24T01:07:54.148 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:07:54.773 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:54.762+0000 7fb26ad21d80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T01:07:55.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:07:55.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:07:55.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:07:55.150 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:07:55.150 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:07:55.150 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:07:55.514 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/58052,v1:127.0.0.1:6819/58052] [v2:127.0.0.1:6820/58052,v1:127.0.0.1:6821/58052] exists,up 1f6ffa51-a8e7-48be-9aae-946bfa091bf1 2020-08-24T01:07:55.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:07:55.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:07:55.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:07:55.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:07:55.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-08-24T01:07:55.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-08-24T01:07:55.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:07:55.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-08-24T01:07:55.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:07:55.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-08-24T01:07:55.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:07:55.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:07:55.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:07:55.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:07:55.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-08-24T01:07:55.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-08-24T01:07:55.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:07:55.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:07:55.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-08-24T01:07:55.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:07:55.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:07:55.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:07:55.529 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:07:55.529 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:07:55.529 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:07:55.529 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:55.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:07:55.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:07:55.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:07:55.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:07:55.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-08-24T01:07:55.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-08-24T01:07:55.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:07:55.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:07:55.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:07:55.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:07:55.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:07:55.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-08-24T01:07:55.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:07:55.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=996e1d1f-98a1-40f5-8c57-35792c16b312 2020-08-24T01:07:55.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 996e1d1f-98a1-40f5-8c57-35792c16b312' 2020-08-24T01:07:55.536 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:07:55.536 INFO:tasks.workunit.client.0.smithi060.stdout:add osd3 996e1d1f-98a1-40f5-8c57-35792c16b312 2020-08-24T01:07:55.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDrEkNfV/RZIBAA9JHuQFlTZ22bJTOTW1t+3Q== 2020-08-24T01:07:55.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDrEkNfV/RZIBAA9JHuQFlTZ22bJTOTW1t+3Q=="}' 2020-08-24T01:07:55.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 996e1d1f-98a1-40f5-8c57-35792c16b312 -i td/osd-backfill-prio/3/new.json 2020-08-24T01:07:56.145 INFO:tasks.workunit.client.0.smithi060.stdout:3 2020-08-24T01:07:56.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-08-24T01:07:56.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDrEkNfV/RZIBAA9JHuQFlTZ22bJTOTW1t+3Q== --osd-uuid 996e1d1f-98a1-40f5-8c57-35792c16b312 2020-08-24T01:07:56.199 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.182+0000 7f4b4748dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:56.200 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.186+0000 7f4b4748dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:56.201 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.190+0000 7f4b4748dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:07:56.201 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.190+0000 7f4b4748dd80 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-08-24T01:07:56.202 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.190+0000 7f4b4748dd80 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-08-24T01:07:56.203 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.190+0000 7f4b4748dd80 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-08-24T01:07:56.245 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.234+0000 7f4b4748dd80 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-08-24T01:07:56.784 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:07:56.782+0000 7f4b4748dd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:07:59.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-08-24T01:07:59.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:07:59.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-08-24T01:07:59.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:07:59.559 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd3 key to auth repository 2020-08-24T01:08:00.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-08-24T01:08:00.039 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:08:00.039 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:08:00.039 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.3 2020-08-24T01:08:00.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:08:00.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:08:00.064 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:00.062+0000 7f7e31959d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:00.066 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:00.062+0000 7f7e31959d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:00.067 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:00.066+0000 7f7e31959d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:00.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-08-24T01:08:00.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:08:00.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-08-24T01:08:00.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:08:00.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:08:00.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:08:00.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:08:00.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:08:00.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:08:00.391 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:08:00.602 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:00.598+0000 7f7e31959d80 -1 Falling back to public interface 2020-08-24T01:08:00.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:08:01.399 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:01.390+0000 7f7e31959d80 -1 osd.3 0 log_to_monitors {default=true} 2020-08-24T01:08:01.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:08:01.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:08:01.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:08:01.780 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:08:01.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:08:01.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:08:02.155 INFO:tasks.workunit.client.0.smithi060.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/58407,v1:127.0.0.1:6827/58407] [v2:127.0.0.1:6828/58407,v1:127.0.0.1:6829/58407] exists,up 996e1d1f-98a1-40f5-8c57-35792c16b312 2020-08-24T01:08:02.155 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:08:02.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:08:02.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:08:02.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:08:02.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-08-24T01:08:02.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-08-24T01:08:02.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:08:02.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-08-24T01:08:02.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:08:02.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-08-24T01:08:02.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-08-24T01:08:02.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:08:02.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:08:02.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:08:02.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-08-24T01:08:02.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-08-24T01:08:02.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:08:02.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-08-24T01:08:02.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-08-24T01:08:02.159 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:08:02.160 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:08:02.160 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:08:02.162 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:08:02.163 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:08:02.163 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:08:02.163 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:08:02.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' 2020-08-24T01:08:02.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:08:02.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:08:02.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:08:02.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-08-24T01:08:02.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-08-24T01:08:02.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:08:02.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:08:02.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:08:02.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:08:02.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:08:02.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-08-24T01:08:02.166 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:08:02.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=631dcb2f-7db5-4b82-9270-78ee24166afb 2020-08-24T01:08:02.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 631dcb2f-7db5-4b82-9270-78ee24166afb' 2020-08-24T01:08:02.166 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:08:02.167 INFO:tasks.workunit.client.0.smithi060.stdout:add osd4 631dcb2f-7db5-4b82-9270-78ee24166afb 2020-08-24T01:08:02.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDyEkNfyzOLChAAPdEcCQjtT+guH2PFrfzkOg== 2020-08-24T01:08:02.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDyEkNfyzOLChAAPdEcCQjtT+guH2PFrfzkOg=="}' 2020-08-24T01:08:02.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 631dcb2f-7db5-4b82-9270-78ee24166afb -i td/osd-backfill-prio/4/new.json 2020-08-24T01:08:02.668 INFO:tasks.workunit.client.0.smithi060.stdout:4 2020-08-24T01:08:02.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-08-24T01:08:02.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDyEkNfyzOLChAAPdEcCQjtT+guH2PFrfzkOg== --osd-uuid 631dcb2f-7db5-4b82-9270-78ee24166afb 2020-08-24T01:08:02.706 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:02.702+0000 7fbeb0f1fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:02.709 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:02.706+0000 7fbeb0f1fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:02.711 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:02.706+0000 7fbeb0f1fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:02.712 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:02.710+0000 7fbeb0f1fd80 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-08-24T01:08:02.712 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:02.710+0000 7fbeb0f1fd80 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-08-24T01:08:02.712 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:02.710+0000 7fbeb0f1fd80 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-08-24T01:08:02.747 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:02.742+0000 7fbeb0f1fd80 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-08-24T01:08:03.124 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:03.122+0000 7fbeb0f1fd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:08:05.951 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-08-24T01:08:05.952 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:08:05.953 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-08-24T01:08:05.954 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:08:05.954 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd4 key to auth repository 2020-08-24T01:08:06.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-08-24T01:08:06.431 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.4 2020-08-24T01:08:06.431 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=5336479e-bba4-4a25-9b05-acc893b3884b --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.50053/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:08:06.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:08:06.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:08:06.432 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:08:06.463 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:06.458+0000 7fd0f808cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:06.464 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:06.462+0000 7fd0f808cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:06.466 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:06.462+0000 7fd0f808cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:08:06.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-08-24T01:08:06.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:08:06.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-08-24T01:08:06.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:08:06.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:08:06.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:08:06.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:08:06.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:08:06.790 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:08:06.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:08:07.005 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:07.002+0000 7fd0f808cd80 -1 Falling back to public interface 2020-08-24T01:08:07.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:08:07.805 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:08:07.790+0000 7fd0f808cd80 -1 osd.4 0 log_to_monitors {default=true} 2020-08-24T01:08:08.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:08:08.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:08:08.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:08:08.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:08:08.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:08:08.171 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:08:08.536 INFO:tasks.workunit.client.0.smithi060.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/58762,v1:127.0.0.1:6835/58762] [v2:127.0.0.1:6836/58762,v1:127.0.0.1:6837/58762] exists,up 631dcb2f-7db5-4b82-9270-78ee24166afb 2020-08-24T01:08:08.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:08:08.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:08:08.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:08:08.538 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-08-24T01:08:08.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:08.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-08-24T01:08:08.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-08-24T01:08:08.991 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test1' already exists 2020-08-24T01:08:08.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:10.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-08-24T01:08:11.014 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 1 size to 2 2020-08-24T01:08:11.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:11.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-08-24T01:08:11.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-08-24T01:08:11.662 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test2' already exists 2020-08-24T01:08:11.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:12.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-08-24T01:08:13.596 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 2 size to 2 2020-08-24T01:08:13.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:13.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-08-24T01:08:13.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-08-24T01:08:14.219 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test3' already exists 2020-08-24T01:08:14.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:15.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-08-24T01:08:16.085 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 3 size to 2 2020-08-24T01:08:16.098 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:16.099 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-08-24T01:08:16.099 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-08-24T01:08:16.705 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test4' already exists 2020-08-24T01:08:16.714 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:17.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-08-24T01:08:18.466 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 4 size to 2 2020-08-24T01:08:18.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:18.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-08-24T01:08:18.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-08-24T01:08:19.240 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test5' already exists 2020-08-24T01:08:19.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:20.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-08-24T01:08:21.122 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 5 size to 2 2020-08-24T01:08:22.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:22.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-08-24T01:08:22.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-08-24T01:08:22.046 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test6' already exists 2020-08-24T01:08:22.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:22.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-08-24T01:08:23.367 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 6 size to 2 2020-08-24T01:08:23.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:23.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-08-24T01:08:23.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-08-24T01:08:24.170 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test7' already exists 2020-08-24T01:08:24.181 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:25.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-08-24T01:08:25.898 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 7 size to 2 2020-08-24T01:08:25.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:25.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-08-24T01:08:25.911 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-08-24T01:08:26.490 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test8' already exists 2020-08-24T01:08:26.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:27.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-08-24T01:08:28.208 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 8 size to 2 2020-08-24T01:08:28.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:28.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-08-24T01:08:28.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-08-24T01:08:28.986 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test9' already exists 2020-08-24T01:08:28.995 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:29.997 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-08-24T01:08:30.790 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 9 size to 2 2020-08-24T01:08:30.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:30.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-08-24T01:08:30.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-08-24T01:08:31.392 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test10' already exists 2020-08-24T01:08:31.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:08:32.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-08-24T01:08:33.315 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 10 size to 2 2020-08-24T01:08:33.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-08-24T01:08:38.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-08-24T01:08:38.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:08:38.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:08:38.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:08:38.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:08:38.331 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:08:38.331 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:08:38.331 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:08:38.332 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:08:38.332 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:08:38.332 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:08:38.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:08:38.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:08:38.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:08:38.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:08:38.551 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:08:38.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:08:38.931 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:08:38.932 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:08:38.932 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:08:38.932 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:08:38.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:08:38.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:38.933 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:08:39.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-08-24T01:08:39.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836494 2020-08-24T01:08:39.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836494' 2020-08-24T01:08:39.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:39.114 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:08:39.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-08-24T01:08:39.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705676 2020-08-24T01:08:39.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836494 1-38654705676' 2020-08-24T01:08:39.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:39.279 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:08:39.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-08-24T01:08:39.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542155 2020-08-24T01:08:39.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836494 1-38654705676 2-60129542155' 2020-08-24T01:08:39.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:39.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:08:39.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378634 2020-08-24T01:08:39.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378634 2020-08-24T01:08:39.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836494 1-38654705676 2-60129542155 3-81604378634' 2020-08-24T01:08:39.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:39.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:08:39.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247817 2020-08-24T01:08:39.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247817 2020-08-24T01:08:39.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836494 1-38654705676 2-60129542155 3-81604378634 4-98784247817' 2020-08-24T01:08:39.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:08:39.749 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836494 2020-08-24T01:08:39.749 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:08:39.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:08:39.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836494 2020-08-24T01:08:39.752 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:08:39.752 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836494 2020-08-24T01:08:39.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-08-24T01:08:39.753 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836494 2020-08-24T01:08:39.753 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:08:40.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836494 -lt 21474836494 2020-08-24T01:08:40.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:08:40.120 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705676 2020-08-24T01:08:40.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:08:40.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:08:40.124 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:08:40.124 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705676 2020-08-24T01:08:40.124 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705676 2020-08-24T01:08:40.124 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-08-24T01:08:40.125 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705676 2020-08-24T01:08:40.125 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:08:40.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705677 -lt 38654705676 2020-08-24T01:08:40.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:08:40.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542155 2020-08-24T01:08:40.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:08:40.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:08:40.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542155 2020-08-24T01:08:40.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:08:40.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542155 2020-08-24T01:08:40.500 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-08-24T01:08:40.505 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542155 2020-08-24T01:08:40.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:08:40.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542155 -lt 60129542155 2020-08-24T01:08:40.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:08:40.869 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378634 2020-08-24T01:08:40.869 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:08:40.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:08:40.872 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:08:40.872 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378634 2020-08-24T01:08:40.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378634 2020-08-24T01:08:40.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378634' 2020-08-24T01:08:40.873 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378634 2020-08-24T01:08:40.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:08:41.345 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378634 -lt 81604378634 2020-08-24T01:08:41.346 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:08:41.346 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247817 2020-08-24T01:08:41.347 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:08:41.348 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:08:41.349 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247817 2020-08-24T01:08:41.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:08:41.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247817 2020-08-24T01:08:41.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247817' 2020-08-24T01:08:41.352 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247817 2020-08-24T01:08:41.352 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:08:41.759 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247817 -lt 98784247817 2020-08-24T01:08:41.760 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:08:41.760 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:08:41.760 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:08:42.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-08-24T01:08:42.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:08:42.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:08:42.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:08:42.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:08:42.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:08:42.239 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:08:42.239 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:08:42.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-08-24T01:08:42.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:08:42.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:08:42.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:08:43.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-08-24T01:08:43.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:08:43.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:08:43.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-08-24T01:08:43.401 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:08:43.401 INFO:tasks.workunit.client.0.smithi060.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:33.804376+0000 0'0 80:14 [2,1] 2 [2,1] 2 0'0 2020-08-24T01:08:31.190535+0000 0'0 2020-08-24T01:08:31.190535+0000 0 2020-08-24T01:08:43.402 INFO:tasks.workunit.client.0.smithi060.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:31.459713+0000 0'0 80:24 [1,2] 1 [1,2] 1 0'0 2020-08-24T01:08:28.701139+0000 0'0 2020-08-24T01:08:28.701139+0000 0 2020-08-24T01:08:43.402 INFO:tasks.workunit.client.0.smithi060.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:29.277529+0000 0'0 80:29 [3,4] 3 [3,4] 3 0'0 2020-08-24T01:08:26.295430+0000 0'0 2020-08-24T01:08:26.295430+0000 0 2020-08-24T01:08:43.402 INFO:tasks.workunit.client.0.smithi060.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:26.808964+0000 0'0 80:33 [1,4] 1 [1,4] 1 0'0 2020-08-24T01:08:23.736724+0000 0'0 2020-08-24T01:08:23.736724+0000 0 2020-08-24T01:08:43.402 INFO:tasks.workunit.client.0.smithi060.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:14.432465+0000 0'0 80:57 [3,1] 3 [3,1] 3 0'0 2020-08-24T01:08:11.451254+0000 0'0 2020-08-24T01:08:11.451254+0000 0 2020-08-24T01:08:43.403 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:11.745356+0000 0'0 80:54 [1,0] 1 [1,0] 1 0'0 2020-08-24T01:08:08.835653+0000 0'0 2020-08-24T01:08:08.835653+0000 0 2020-08-24T01:08:43.403 INFO:tasks.workunit.client.0.smithi060.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:16.963016+0000 0'0 80:47 [1,2] 1 [1,2] 1 0'0 2020-08-24T01:08:14.028831+0000 0'0 2020-08-24T01:08:14.028831+0000 0 2020-08-24T01:08:43.403 INFO:tasks.workunit.client.0.smithi060.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:18.920909+0000 0'0 79:39 [4,0] 4 [4,0] 4 0'0 2020-08-24T01:08:16.539720+0000 0'0 2020-08-24T01:08:16.539720+0000 0 2020-08-24T01:08:43.403 INFO:tasks.workunit.client.0.smithi060.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:21.317345+0000 0'0 80:43 [3,4] 3 [3,4] 3 0'0 2020-08-24T01:08:18.872655+0000 0'0 2020-08-24T01:08:18.872655+0000 0 2020-08-24T01:08:43.403 INFO:tasks.workunit.client.0.smithi060.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-08-24T01:08:24.960035+0000 0'0 79:37 [4,2] 4 [4,2] 4 0'0 2020-08-24T01:08:21.469382+0000 0'0 2020-08-24T01:08:21.469382+0000 0 2020-08-24T01:08:43.404 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:08:43.404 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:08:43.404 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:08:43.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-08-24T01:08:43.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-08-24T01:08:43.420 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-08-24T01:08:43.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-08-24T01:08:43.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-08-24T01:08:43.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-08-24T01:08:43.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-08-24T01:08:43.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-08-24T01:08:43.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-08-24T01:08:43.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-08-24T01:08:43.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-08-24T01:08:43.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-08-24T01:08:43.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-08-24T01:08:43.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:43.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-08-24T01:08:43.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-08-24T01:08:43.781 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:08:43.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-08-24T01:08:43.782 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:08:43.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-08-24T01:08:43.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-08-24T01:08:43.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-08-24T01:08:43.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-08-24T01:08:43.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-08-24T01:08:43.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-08-24T01:08:43.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-08-24T01:08:43.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:43.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-08-24T01:08:43.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-08-24T01:08:44.160 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:08:44.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-08-24T01:08:44.161 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:08:44.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-08-24T01:08:44.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-08-24T01:08:44.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-08-24T01:08:44.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-08-24T01:08:44.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:44.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-08-24T01:08:44.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-08-24T01:08:44.533 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:08:44.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-08-24T01:08:44.536 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:08:44.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-08-24T01:08:44.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-08-24T01:08:44.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-08-24T01:08:44.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-08-24T01:08:44.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-08-24T01:08:44.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-08-24T01:08:44.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-08-24T01:08:44.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:44.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-08-24T01:08:44.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-08-24T01:08:44.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:08:44.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-08-24T01:08:44.886 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:08:44.888 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-08-24T01:08:44.888 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-08-24T01:08:44.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-08-24T01:08:44.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-08-24T01:08:44.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:44.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-08-24T01:08:44.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-08-24T01:08:45.252 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:08:45.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-08-24T01:08:45.253 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:08:45.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-08-24T01:08:45.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-08-24T01:08:45.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-08-24T01:08:45.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-08-24T01:08:45.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:45.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-08-24T01:08:45.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-08-24T01:08:45.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:08:45.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-08-24T01:08:45.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:08:45.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-08-24T01:08:45.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-08-24T01:08:45.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-08-24T01:08:45.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-08-24T01:08:45.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:45.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-08-24T01:08:45.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-08-24T01:08:46.028 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-08-24T01:08:46.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-08-24T01:08:46.030 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-08-24T01:08:46.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-08-24T01:08:46.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-08-24T01:08:46.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-08-24T01:08:46.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-08-24T01:08:46.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-08-24T01:08:46.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-08-24T01:08:46.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-08-24T01:08:46.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-08-24T01:08:46.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-08-24T01:08:46.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-08-24T01:08:46.035 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-08-24T01:08:46.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:46.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-08-24T01:08:46.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:46.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-08-24T01:08:46.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-08-24T01:08:46.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-08-24T01:08:46.037 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-08-24T01:08:46.574 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test2' does not exist 2020-08-24T01:08:46.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:46.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-08-24T01:08:46.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:46.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-08-24T01:08:46.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-08-24T01:08:46.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-08-24T01:08:46.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-08-24T01:08:47.083 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test4' does not exist 2020-08-24T01:08:47.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:47.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-08-24T01:08:47.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-08-24T01:08:47.094 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-08-24T01:08:47.094 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-08-24T01:08:47.565 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test5' does not exist 2020-08-24T01:08:47.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:47.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-08-24T01:08:47.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-08-24T01:08:47.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-08-24T01:08:47.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-08-24T01:08:48.076 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test6' does not exist 2020-08-24T01:08:48.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:48.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-08-24T01:08:48.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:48.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-08-24T01:08:48.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-08-24T01:08:48.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-08-24T01:08:48.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-08-24T01:08:48.620 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test8' does not exist 2020-08-24T01:08:48.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:48.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-08-24T01:08:48.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-08-24T01:08:48.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-08-24T01:08:48.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-08-24T01:08:49.148 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test9' does not exist 2020-08-24T01:08:49.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-08-24T01:08:49.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-08-24T01:08:49.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-08-24T01:08:49.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-08-24T01:08:49.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-08-24T01:08:49.789 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test10' does not exist 2020-08-24T01:08:49.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-08-24T01:08:50.569 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 3 size to 1 2020-08-24T01:08:50.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-08-24T01:08:51.240 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 7 size to 1 2020-08-24T01:08:51.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-08-24T01:08:51.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:08:51.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:08:51.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:08:51.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:08:51.258 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:08:51.258 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:08:51.258 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:08:51.258 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:08:51.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:08:51.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:08:51.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:08:51.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:08:51.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:08:51.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:08:51.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:08:51.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:08:51.902 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:08:51.902 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:08:51.903 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:08:51.903 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:08:51.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:08:51.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:51.904 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:08:52.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836498 2020-08-24T01:08:52.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836498 2020-08-24T01:08:52.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836498' 2020-08-24T01:08:52.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:52.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:08:52.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705681 2020-08-24T01:08:52.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705681 2020-08-24T01:08:52.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836498 1-38654705681' 2020-08-24T01:08:52.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:52.224 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:08:52.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542160 2020-08-24T01:08:52.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542160 2020-08-24T01:08:52.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836498 1-38654705681 2-60129542160' 2020-08-24T01:08:52.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:52.379 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:08:52.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378638 2020-08-24T01:08:52.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378638 2020-08-24T01:08:52.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836498 1-38654705681 2-60129542160 3-81604378638' 2020-08-24T01:08:52.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:08:52.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:08:52.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247821 2020-08-24T01:09:12.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247821 2020-08-24T01:09:12.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836498 1-38654705681 2-60129542160 3-81604378638 4-98784247821' 2020-08-24T01:09:12.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:09:12.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836498 2020-08-24T01:09:12.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:09:12.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:09:12.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836498 2020-08-24T01:09:12.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:09:12.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836498 2020-08-24T01:09:12.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-08-24T01:09:12.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:09:12.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836496 -lt 21474836498 2020-08-24T01:09:12.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:09:12.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:09:12.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:09:12.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836499 -lt 21474836498 2020-08-24T01:09:12.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:09:12.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705681 2020-08-24T01:09:12.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:09:12.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:09:12.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705681 2020-08-24T01:09:12.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:09:12.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705681 2020-08-24T01:09:12.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705681' 2020-08-24T01:09:12.492 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:09:12.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705681 -lt 38654705681 2020-08-24T01:09:12.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:09:12.492 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542160 2020-08-24T01:09:12.492 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:09:12.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:09:12.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:09:12.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542160 2020-08-24T01:09:12.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542160 2020-08-24T01:09:12.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542160' 2020-08-24T01:09:12.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:09:12.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542160 -lt 60129542160 2020-08-24T01:09:12.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:09:12.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378638 2020-08-24T01:09:12.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:09:12.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:09:12.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378638 2020-08-24T01:09:12.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:09:12.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378638 2020-08-24T01:09:12.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2020-08-24T01:09:12.496 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:09:12.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378639 -lt 81604378638 2020-08-24T01:09:12.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:09:12.497 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247821 2020-08-24T01:09:12.497 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:09:12.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:09:12.497 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247821 2020-08-24T01:09:12.497 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:09:12.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247821 2020-08-24T01:09:12.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247821' 2020-08-24T01:09:12.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:09:12.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247821 -lt 98784247821 2020-08-24T01:09:12.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:09:12.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:09:12.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:09:12.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-08-24T01:09:12.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:09:12.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:09:12.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:09:12.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:09:12.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:09:12.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:09:12.501 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:09:12.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-08-24T01:09:12.501 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:09:12.501 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:09:12.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:09:12.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-08-24T01:09:12.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:09:12.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:09:12.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-08-24T01:09:12.503 INFO:tasks.workunit.client.0.smithi060.stderr:10+0 records in 2020-08-24T01:09:12.503 INFO:tasks.workunit.client.0.smithi060.stderr:10+0 records out 2020-08-24T01:09:12.503 INFO:tasks.workunit.client.0.smithi060.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0566705 s, 185 MB/s 2020-08-24T01:09:12.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-08-24T01:09:12.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-08-24T01:09:12.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-08-24T01:09:12.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.509 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.510 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-08-24T01:09:12.515 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836498 2020-08-24T01:09:12.515 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705681 2020-08-24T01:09:12.515 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542160 2020-08-24T01:09:12.515 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378638 2020-08-24T01:09:12.515 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247821 2020-08-24T01:09:12.831 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:12.831 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-08-24T01:09:13.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:13.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-08-24T01:09:14.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:14.138 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-08-24T01:09:14.759 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:14.759 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-08-24T01:09:15.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:15.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-08-24T01:09:16.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:16.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-08-24T01:09:16.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:16.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-08-24T01:09:17.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:17.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-08-24T01:09:17.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:17.986 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-08-24T01:09:18.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:18.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-08-24T01:09:19.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:19.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-08-24T01:09:20.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:20.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-08-24T01:09:20.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:20.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-08-24T01:09:21.345 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:21.346 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-08-24T01:09:21.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:21.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-08-24T01:09:22.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:22.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-08-24T01:09:23.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:23.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-08-24T01:09:23.981 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:24.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-08-24T01:09:24.991 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:24.991 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-08-24T01:09:25.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:25.294 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-08-24T01:09:25.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:25.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-08-24T01:09:26.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:26.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-08-24T01:09:27.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:27.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-08-24T01:09:28.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:28.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-08-24T01:09:28.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:28.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-08-24T01:09:29.372 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:29.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-08-24T01:09:30.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:30.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-08-24T01:09:30.721 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-08-24T01:09:30.722 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-08-24T01:09:30.722 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-08-24T01:09:30.722 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-08-24T01:09:30.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:30.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-08-24T01:09:31.124 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:31.125 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-08-24T01:09:31.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:31.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-08-24T01:09:31.907 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:31.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-08-24T01:09:32.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:32.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-08-24T01:09:32.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:32.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-08-24T01:09:33.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:33.075 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-08-24T01:09:33.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:33.463 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-08-24T01:09:33.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:33.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-08-24T01:09:34.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:34.281 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-08-24T01:09:34.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:34.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-08-24T01:09:35.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:35.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-08-24T01:09:35.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:35.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-08-24T01:09:35.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:35.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-08-24T01:09:36.265 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:36.265 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-08-24T01:09:36.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:36.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-08-24T01:09:37.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:37.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-08-24T01:09:37.444 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:37.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-08-24T01:09:37.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:37.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-08-24T01:09:38.285 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:38.285 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-08-24T01:09:38.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:38.707 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-08-24T01:09:39.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:39.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-08-24T01:09:39.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:39.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-08-24T01:09:39.842 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:39.843 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-08-24T01:09:40.263 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:40.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-08-24T01:09:40.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:40.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-08-24T01:09:41.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:41.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-08-24T01:09:41.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:41.402 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-08-24T01:09:41.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:41.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-08-24T01:09:42.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:42.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-08-24T01:09:42.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:42.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-08-24T01:09:43.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:43.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-08-24T01:09:43.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:43.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-08-24T01:09:43.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:43.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-08-24T01:09:44.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:44.274 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-08-24T01:09:44.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:44.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-08-24T01:09:45.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:45.024 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-08-24T01:09:45.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:45.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-08-24T01:09:45.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:45.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-08-24T01:09:46.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:46.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-08-24T01:09:46.654 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:46.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-08-24T01:09:47.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:47.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-08-24T01:09:47.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:47.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-08-24T01:09:47.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:47.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-08-24T01:09:48.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:48.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-08-24T01:09:48.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:48.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-08-24T01:09:48.979 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:48.980 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-08-24T01:09:49.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:49.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-08-24T01:09:49.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:49.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-08-24T01:09:50.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:50.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-08-24T01:09:50.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-08-24T01:09:50.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-08-24T01:09:50.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-08-24T01:09:50.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-08-24T01:09:50.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:50.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-08-24T01:09:50.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:50.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-08-24T01:09:51.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:51.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-08-24T01:09:51.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:51.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-08-24T01:09:52.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:52.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-08-24T01:09:52.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:52.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-08-24T01:09:52.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:52.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-08-24T01:09:53.168 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:53.169 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-08-24T01:09:53.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:53.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-08-24T01:09:53.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:53.915 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-08-24T01:09:54.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:54.306 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-08-24T01:09:54.731 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:54.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-08-24T01:09:55.102 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-08-24T01:09:58.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:58.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-08-24T01:09:59.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:59.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-08-24T01:09:59.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:59.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-08-24T01:09:59.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:09:59.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-08-24T01:10:00.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:00.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-08-24T01:10:00.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:00.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-08-24T01:10:00.962 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:00.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-08-24T01:10:01.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:01.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-08-24T01:10:01.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:01.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-08-24T01:10:02.143 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:02.144 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-08-24T01:10:02.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:02.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-08-24T01:10:02.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:02.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-08-24T01:10:03.292 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:03.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-08-24T01:10:03.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:03.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-08-24T01:10:04.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:04.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-08-24T01:10:04.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:04.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-08-24T01:10:04.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:04.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-08-24T01:10:05.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:05.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-08-24T01:10:05.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:05.566 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-08-24T01:10:05.962 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:05.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-08-24T01:10:06.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:06.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-08-24T01:10:06.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:06.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-08-24T01:10:07.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:07.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-08-24T01:10:07.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:07.529 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-08-24T01:10:07.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:07.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-08-24T01:10:08.291 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:08.292 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-08-24T01:10:08.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:08.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-08-24T01:10:09.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:09.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-08-24T01:10:09.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-08-24T01:10:09.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-08-24T01:10:09.798 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-08-24T01:10:09.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-08-24T01:10:09.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-08-24T01:10:09.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-08-24T01:10:09.801 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-08-24T01:10:09.802 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-08-24T01:10:09.802 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-08-24T01:10:09.802 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-08-24T01:10:09.803 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-08-24T01:10:09.803 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-08-24T01:10:10.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-08-24T01:10:10.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-08-24T01:10:10.192 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-08-24T01:10:10.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-08-24T01:10:10.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-08-24T01:10:10.875 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:10:10.875 INFO:tasks.workunit.client.0.smithi060.stdout:7.0 49 0 0 0 0 513802240 0 0 47 47 active+clean 2020-08-24T01:08:51.793945+0000 93'147 93:195 [1] 1 [1] 1 0'0 2020-08-24T01:08:23.736724+0000 0'0 2020-08-24T01:08:23.736724+0000 0 2020-08-24T01:10:10.876 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-08-24T01:08:11.745356+0000 93'150 93:212 [1,0] 1 [1,0] 1 0'0 2020-08-24T01:08:08.835653+0000 0'0 2020-08-24T01:08:08.835653+0000 0 2020-08-24T01:10:10.876 INFO:tasks.workunit.client.0.smithi060.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-08-24T01:08:51.848197+0000 93'150 93:212 [1] 1 [1] 1 0'0 2020-08-24T01:08:14.028831+0000 0'0 2020-08-24T01:08:14.028831+0000 0 2020-08-24T01:10:10.876 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:10:10.876 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:10:10.877 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:10:10.888 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-08-24T01:10:10.888 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-08-24T01:10:11.552 INFO:tasks.workunit.client.0.smithi060.stderr:nobackfill is set 2020-08-24T01:10:11.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-08-24T01:10:12.241 INFO:tasks.workunit.client.0.smithi060.stderr:noout is set 2020-08-24T01:10:12.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-08-24T01:10:13.037 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 7 size to 2 2020-08-24T01:10:13.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-08-24T01:10:15.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:10:15.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:10:15.054 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:10:15.055 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:10:15.055 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:10:15.055 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:10:15.055 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:10:15.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:10:15.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:10:15.159 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:10:15.159 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:10:15.160 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:15.160 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:15.160 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:15.161 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:15.161 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:15.161 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:10:15.162 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:15.162 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:15.162 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:15.162 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:15.163 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:15.163 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:10:15.169 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-08-24T01:10:15.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-08-24T01:10:15.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-08-24T01:10:15.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-08-24T01:10:15.531 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 1 = 10 ']' 2020-08-24T01:10:15.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-08-24T01:10:17.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-08-24T01:10:17.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-08-24T01:10:17.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-08-24T01:10:17.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:168: TEST_backfill_priority: '[' 2 = 10 ']' 2020-08-24T01:10:17.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:172: TEST_backfill_priority: sleep 2 2020-08-24T01:10:19.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-08-24T01:10:19.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-08-24T01:10:19.905 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-08-24T01:10:20.290 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-08-24T01:10:20.291 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-08-24T01:10:20.291 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:10:20.291 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:10:20.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:10:20.679 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:10:20.679 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:10:20.679 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:10:20.679 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:10:20.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:10:20.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:20.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:10:20.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836518 2020-08-24T01:10:20.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836518 2020-08-24T01:10:20.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836518' 2020-08-24T01:10:20.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:20.851 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:10:21.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705701 2020-08-24T01:10:21.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705701 2020-08-24T01:10:21.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836518 1-38654705701' 2020-08-24T01:10:21.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:21.002 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:10:21.155 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542180 2020-08-24T01:10:21.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542180 2020-08-24T01:10:21.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542180' 2020-08-24T01:10:21.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:21.157 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:10:21.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378658 2020-08-24T01:10:21.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378658 2020-08-24T01:10:21.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542180 3-81604378658' 2020-08-24T01:10:21.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:21.331 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:10:21.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247841 2020-08-24T01:10:21.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247841 2020-08-24T01:10:21.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836518 1-38654705701 2-60129542180 3-81604378658 4-98784247841' 2020-08-24T01:10:21.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:21.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836518 2020-08-24T01:10:21.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:21.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:10:21.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836518 2020-08-24T01:10:21.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:21.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836518 2020-08-24T01:10:21.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836518' 2020-08-24T01:10:21.491 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836518 2020-08-24T01:10:21.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:21.849 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836516 -lt 21474836518 2020-08-24T01:10:21.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:10:22.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:10:22.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:23.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836518 -lt 21474836518 2020-08-24T01:10:23.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:23.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705701 2020-08-24T01:10:23.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:23.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:10:23.225 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705701 2020-08-24T01:10:23.225 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:23.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705701 2020-08-24T01:10:23.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705701' 2020-08-24T01:10:23.226 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:10:23.226 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705701 2020-08-24T01:10:23.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705701 -lt 38654705701 2020-08-24T01:10:23.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:23.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542180 2020-08-24T01:10:23.601 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:23.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:10:23.602 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:23.603 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542180 2020-08-24T01:10:23.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542180 2020-08-24T01:10:23.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2020-08-24T01:10:23.608 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:10:23.609 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542180 2020-08-24T01:10:23.971 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542180 -lt 60129542180 2020-08-24T01:10:23.971 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:23.972 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378658 2020-08-24T01:10:23.972 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:23.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:10:23.974 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:23.974 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378658 2020-08-24T01:10:23.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378658 2020-08-24T01:10:23.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378658' 2020-08-24T01:10:23.976 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:10:23.976 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378658 2020-08-24T01:10:24.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378659 -lt 81604378658 2020-08-24T01:10:24.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:24.492 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247841 2020-08-24T01:10:24.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:24.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:10:24.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247841 2020-08-24T01:10:24.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:24.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247841 2020-08-24T01:10:24.503 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247841' 2020-08-24T01:10:24.503 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:10:24.503 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247841 2020-08-24T01:10:24.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247841 -lt 98784247841 2020-08-24T01:10:24.864 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:10:24.864 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:10:24.864 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:10:24.865 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:10:24.865 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:10:24.865 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:10:24.865 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:10:24.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:10:24.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:10:24.955 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:10:24.955 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:10:24.956 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:24.956 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:24.956 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:24.956 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:10:24.956 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:24.957 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:10:24.957 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:10:24.957 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:24.957 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:24.958 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:24.958 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:24.958 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:10:24.959 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:24.959 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:24.959 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:24.959 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:24.959 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:24.960 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:10:24.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-08-24T01:10:25.484 INFO:tasks.workunit.client.0.smithi060.stderr:osd.0 is already out. 2020-08-24T01:10:25.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-08-24T01:10:27.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-08-24T01:10:29.872 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:10:29.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:10:29.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:10:29.878 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:10:29.878 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:10:29.878 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:10:29.878 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:10:29.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:10:29.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:29.879 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:10:29.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836522 2020-08-24T01:10:29.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836522 2020-08-24T01:10:29.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836522' 2020-08-24T01:10:29.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:29.880 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:10:29.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705704 2020-08-24T01:10:29.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705704 2020-08-24T01:10:29.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836522 1-38654705704' 2020-08-24T01:10:29.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:29.881 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:10:29.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542183 2020-08-24T01:10:29.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542183 2020-08-24T01:10:29.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836522 1-38654705704 2-60129542183' 2020-08-24T01:10:29.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:29.882 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:10:29.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378662 2020-08-24T01:10:29.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378662 2020-08-24T01:10:29.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836522 1-38654705704 2-60129542183 3-81604378662' 2020-08-24T01:10:29.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:29.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:10:29.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247844 2020-08-24T01:10:29.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247844 2020-08-24T01:10:29.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836522 1-38654705704 2-60129542183 3-81604378662 4-98784247844' 2020-08-24T01:10:29.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:29.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836522 2020-08-24T01:10:29.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:29.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:10:29.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836522 2020-08-24T01:10:29.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:29.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836522 2020-08-24T01:10:29.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2020-08-24T01:10:29.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:29.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836520 -lt 21474836522 2020-08-24T01:10:29.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:10:29.895 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836522 2020-08-24T01:10:30.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:10:30.061 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:30.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836522 -lt 21474836522 2020-08-24T01:10:30.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:30.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705704 2020-08-24T01:10:30.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:30.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:10:30.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705704 2020-08-24T01:10:30.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:30.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705704 2020-08-24T01:10:30.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705704' 2020-08-24T01:10:30.453 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:10:30.454 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705704 2020-08-24T01:10:30.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705705 -lt 38654705704 2020-08-24T01:10:30.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:30.815 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542183 2020-08-24T01:10:30.815 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:30.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:10:30.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542183 2020-08-24T01:10:30.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:30.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542183 2020-08-24T01:10:30.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542183' 2020-08-24T01:10:30.820 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:10:30.820 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542183 2020-08-24T01:10:31.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542183 -lt 60129542183 2020-08-24T01:10:31.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:31.202 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378662 2020-08-24T01:10:31.202 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:31.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:10:31.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378662 2020-08-24T01:10:31.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:31.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378662 2020-08-24T01:10:31.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378662' 2020-08-24T01:10:31.207 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378662 2020-08-24T01:10:31.207 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:10:31.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378662 -lt 81604378662 2020-08-24T01:10:31.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:31.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247844 2020-08-24T01:10:31.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:31.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:10:31.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247844 2020-08-24T01:10:31.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:31.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247844 2020-08-24T01:10:31.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247844' 2020-08-24T01:10:31.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:10:31.585 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247844 2020-08-24T01:10:31.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247845 -lt 98784247844 2020-08-24T01:10:31.969 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:10:31.969 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:10:31.969 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:10:31.970 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:10:31.970 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:10:31.970 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:10:31.971 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:10:31.971 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:10:31.971 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:10:32.057 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:10:32.057 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:10:32.057 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:32.058 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:32.058 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:10:32.058 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:32.058 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:10:32.058 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:32.059 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:32.059 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:10:32.059 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:10:32.059 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:32.059 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:32.060 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:32.060 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:32.060 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:10:32.060 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:10:32.060 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:32.060 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:10:32.061 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:10:32.061 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:32.061 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:32.061 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:32.061 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:32.062 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:10:32.062 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:32.062 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:32.062 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:32.062 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:32.063 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:32.063 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:10:32.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-08-24T01:10:32.394 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:10:32.395 INFO:tasks.workunit.client.0.smithi060.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-08-24T01:10:20.181489+0000 93'150 104:217 [1,4] 1 [1] 1 0'0 2020-08-24T01:08:23.736724+0000 0'0 2020-08-24T01:08:23.736724+0000 0 2020-08-24T01:10:32.395 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-08-24T01:10:27.094974+0000 93'150 105:229 [1,2] 1 [1,0] 1 0'0 2020-08-24T01:08:08.835653+0000 0'0 2020-08-24T01:08:08.835653+0000 0 2020-08-24T01:10:32.395 INFO:tasks.workunit.client.0.smithi060.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-08-24T01:08:51.848197+0000 93'150 104:221 [1] 1 [1] 1 0'0 2020-08-24T01:08:14.028831+0000 0'0 2020-08-24T01:08:14.028831+0000 0 2020-08-24T01:10:32.395 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:10:32.395 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:10:32.396 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:10:32.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-08-24T01:10:33.100 INFO:tasks.workunit.client.0.smithi060.stderr:set pool 3 size to 2 2020-08-24T01:10:33.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-08-24T01:10:35.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-08-24T01:10:35.124 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:10:35.124 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:10:35.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:10:35.482 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:10:35.483 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:10:35.483 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:10:35.483 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:10:35.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:10:35.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:35.484 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:10:35.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836525 2020-08-24T01:10:35.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836525 2020-08-24T01:10:35.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836525' 2020-08-24T01:10:35.674 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:35.674 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:10:35.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705708 2020-08-24T01:10:35.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705708 2020-08-24T01:10:35.838 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836525 1-38654705708' 2020-08-24T01:10:35.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:35.839 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:10:36.013 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542186 2020-08-24T01:10:36.014 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542186 2020-08-24T01:10:36.014 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836525 1-38654705708 2-60129542186' 2020-08-24T01:10:36.014 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:36.015 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:10:36.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378665 2020-08-24T01:10:36.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378665 2020-08-24T01:10:36.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836525 1-38654705708 2-60129542186 3-81604378665' 2020-08-24T01:10:36.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:36.174 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:10:36.347 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247848 2020-08-24T01:10:36.347 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247848 2020-08-24T01:10:36.347 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836525 1-38654705708 2-60129542186 3-81604378665 4-98784247848' 2020-08-24T01:10:36.348 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:36.348 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836525 2020-08-24T01:10:36.348 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:36.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:10:36.349 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836525 2020-08-24T01:10:36.349 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:36.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836525 2020-08-24T01:10:36.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836525' 2020-08-24T01:10:36.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:36.351 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836525 2020-08-24T01:10:36.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836525 -lt 21474836525 2020-08-24T01:10:36.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:36.712 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705708 2020-08-24T01:10:36.712 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:36.714 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:10:36.715 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705708 2020-08-24T01:10:36.716 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:36.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705708 2020-08-24T01:10:36.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705708' 2020-08-24T01:10:36.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:10:36.720 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705708 2020-08-24T01:10:37.097 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705708 -lt 38654705708 2020-08-24T01:10:37.097 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:37.100 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542186 2020-08-24T01:10:37.100 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:37.100 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:10:37.101 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:37.101 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542186 2020-08-24T01:10:37.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542186 2020-08-24T01:10:37.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542186' 2020-08-24T01:10:37.103 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:10:37.104 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542186 2020-08-24T01:10:37.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542184 -lt 60129542186 2020-08-24T01:10:37.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:10:38.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:10:38.481 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:10:38.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542187 -lt 60129542186 2020-08-24T01:10:38.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:38.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378665 2020-08-24T01:10:38.857 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:38.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:10:38.858 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378665 2020-08-24T01:10:38.859 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:38.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378665 2020-08-24T01:10:38.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378665' 2020-08-24T01:10:38.860 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378665 2020-08-24T01:10:38.861 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:10:39.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378666 -lt 81604378665 2020-08-24T01:10:39.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:39.238 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247848 2020-08-24T01:10:39.239 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:39.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:10:39.241 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247848 2020-08-24T01:10:39.241 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:39.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247848 2020-08-24T01:10:39.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247848' 2020-08-24T01:10:39.243 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247848 2020-08-24T01:10:39.243 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:10:39.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247848 -lt 98784247848 2020-08-24T01:10:39.607 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:10:39.608 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:10:39.608 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:10:39.608 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:10:39.609 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:10:39.609 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:10:39.609 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:10:39.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:10:39.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:10:39.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:39.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-08-24T01:10:39.718 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:10:39.719 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:10:39.719 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:39.719 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:39.719 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:10:39.719 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:39.720 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:10:39.720 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:39.720 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:39.721 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:10:39.721 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:10:39.721 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:39.722 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:39.722 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:39.722 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:39.722 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:39.722 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:10:39.723 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:39.723 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:39.723 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:10:39.723 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:10:39.723 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:39.723 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:39.724 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:39.724 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:39.724 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:10:39.724 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:10:39.724 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:39.725 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:10:39.725 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:10:39.725 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:39.725 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:39.725 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:39.735 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:39.735 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:10:39.735 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:39.735 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:39.736 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:39.736 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:39.736 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:39.736 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:10:40.044 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:10:40.045 INFO:tasks.workunit.client.0.smithi060.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-08-24T01:10:20.181489+0000 93'150 108:220 [1,4] 1 [1] 1 0'0 2020-08-24T01:08:23.736724+0000 0'0 2020-08-24T01:08:23.736724+0000 0 2020-08-24T01:10:40.045 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-08-24T01:10:27.094974+0000 93'150 108:232 [1,2] 1 [1,0] 1 0'0 2020-08-24T01:08:08.835653+0000 0'0 2020-08-24T01:08:08.835653+0000 0 2020-08-24T01:10:40.046 INFO:tasks.workunit.client.0.smithi060.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-08-24T01:10:33.868782+0000 93'150 109:231 [1,2] 1 [1] 1 0'0 2020-08-24T01:08:14.028831+0000 0'0 2020-08-24T01:08:14.028831+0000 0 2020-08-24T01:10:40.046 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:10:40.046 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:10:40.047 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:10:40.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:40.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-08-24T01:10:40.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-08-24T01:10:40.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-08-24T01:10:40.063 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:40.064 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-08-24T01:10:40.069 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-08-24T01:10:40.070 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-08-24T01:10:40.070 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-08-24T01:10:40.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:40.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-08-24T01:10:40.077 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-08-24T01:10:40.077 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-08-24T01:10:40.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-08-24T01:10:40.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-08-24T01:10:40.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-08-24T01:10:40.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-08-24T01:10:40.447 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-08-24T01:10:40.447 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-08-24T01:10:42.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:10:42.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:10:42.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:10:42.451 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:10:42.452 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:10:42.452 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:10:42.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:10:42.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:10:42.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:10:42.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:42.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:42.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-08-24T01:10:42.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-08-24T01:10:42.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-08-24T01:10:42.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-08-24T01:10:42.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:10:42.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:10:42.586 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:10:42.586 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:10:42.586 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:42.586 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:42.586 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:10:42.587 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:42.587 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:10:42.587 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:42.587 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:42.587 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:10:42.588 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:10:42.588 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:42.588 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:42.588 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:42.588 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:42.589 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:42.589 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 254, 2020-08-24T01:10:42.589 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:42.589 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:42.589 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:10:42.589 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:10:42.590 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:42.591 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:42.591 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:42.593 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:42.593 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:10:42.593 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:10:42.594 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:42.594 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:10:42.594 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:10:42.594 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:42.594 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:42.595 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:42.595 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:42.595 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:10:42.595 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:42.595 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:42.596 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:42.596 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:42.596 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:42.596 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:10:42.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:10:42.943 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:10:42.943 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:10:42.943 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:10:42.943 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:10:42.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:10:42.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:42.944 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:10:43.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836529 2020-08-24T01:10:43.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836529 2020-08-24T01:10:43.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836529' 2020-08-24T01:10:43.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:43.117 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:10:43.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705711 2020-08-24T01:10:43.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705711 2020-08-24T01:10:43.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836529 1-38654705711' 2020-08-24T01:10:43.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:43.280 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:10:43.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542190 2020-08-24T01:10:43.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542190 2020-08-24T01:10:43.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836529 1-38654705711 2-60129542190' 2020-08-24T01:10:43.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:43.451 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:10:43.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378669 2020-08-24T01:10:43.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378669 2020-08-24T01:10:43.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836529 1-38654705711 2-60129542190 3-81604378669' 2020-08-24T01:10:43.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:43.616 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:10:43.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247851 2020-08-24T01:10:43.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247851 2020-08-24T01:10:43.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836529 1-38654705711 2-60129542190 3-81604378669 4-98784247851' 2020-08-24T01:10:43.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:43.770 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836529 2020-08-24T01:10:43.771 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:43.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:10:43.772 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836529 2020-08-24T01:10:43.773 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:43.774 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836529 2020-08-24T01:10:43.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836529 2020-08-24T01:10:43.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836529' 2020-08-24T01:10:43.775 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:44.144 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836529 -lt 21474836529 2020-08-24T01:10:44.148 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:44.149 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705711 2020-08-24T01:10:44.149 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:44.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:10:44.150 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705711 2020-08-24T01:10:44.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:44.151 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705711 2020-08-24T01:10:44.151 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705711' 2020-08-24T01:10:44.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:10:44.152 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705711 2020-08-24T01:10:44.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705711 -lt 38654705711 2020-08-24T01:10:44.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:44.577 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542190 2020-08-24T01:10:44.577 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:44.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:10:44.579 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542190 2020-08-24T01:10:44.579 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:44.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542190 2020-08-24T01:10:44.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542190' 2020-08-24T01:10:44.581 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542190 2020-08-24T01:10:44.582 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:10:44.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542190 -lt 60129542190 2020-08-24T01:10:44.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:44.936 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378669 2020-08-24T01:10:44.937 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:44.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:10:44.938 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378669 2020-08-24T01:10:44.939 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:44.940 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378669 2020-08-24T01:10:44.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378669 2020-08-24T01:10:44.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378669' 2020-08-24T01:10:44.941 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:10:45.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378669 -lt 81604378669 2020-08-24T01:10:45.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:45.306 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247851 2020-08-24T01:10:45.306 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:45.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:10:45.308 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247851 2020-08-24T01:10:45.309 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:45.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247851 2020-08-24T01:10:45.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247851' 2020-08-24T01:10:45.310 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247851 2020-08-24T01:10:45.311 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:10:45.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247851 -lt 98784247851 2020-08-24T01:10:45.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-08-24T01:10:46.044 INFO:tasks.workunit.client.0.smithi060.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-08-24T01:10:46.045 INFO:tasks.workunit.client.0.smithi060.stderr: 2020-08-24T01:10:46.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-08-24T01:10:48.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:10:48.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:10:48.057 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:10:48.059 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:10:48.060 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:10:48.061 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:10:48.061 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:10:48.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:10:48.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:10:48.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:48.171 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:48.171 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-08-24T01:10:48.172 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:10:48.172 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:10:48.172 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:48.173 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:48.173 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:10:48.173 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:48.173 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:10:48.173 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:48.174 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:48.174 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:10:48.174 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:10:48.174 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:48.176 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:48.176 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:48.176 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:48.176 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:48.176 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:10:48.177 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:48.177 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:48.177 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:10:48.177 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:10:48.178 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:48.179 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:48.179 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:48.179 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:48.179 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:10:48.180 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:10:48.180 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:48.180 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:10:48.180 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:10:48.180 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:48.181 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:48.181 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:48.181 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:48.181 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:10:48.181 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:48.182 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:48.182 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:48.182 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:48.182 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:48.182 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:10:48.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-08-24T01:10:48.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-08-24T01:10:48.184 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:48.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-08-24T01:10:48.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-08-24T01:10:48.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-08-24T01:10:48.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-08-24T01:10:48.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:10:48.186 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-08-24T01:10:48.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-08-24T01:10:48.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-08-24T01:10:48.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-08-24T01:10:48.549 INFO:tasks.workunit.client.0.smithi060.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-08-24T01:10:48.549 INFO:tasks.workunit.client.0.smithi060.stderr: 2020-08-24T01:10:48.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-08-24T01:10:53.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:10:53.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:10:53.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:10:53.562 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:10:53.562 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:10:53.562 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:10:53.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:10:53.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:10:53.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:10:53.662 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:10:53.662 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:10:53.663 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:53.663 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:53.663 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:10:53.663 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:53.664 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:10:53.664 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:53.664 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:53.664 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:10:53.664 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:10:53.665 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:53.665 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:53.665 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:53.665 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:53.665 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:53.666 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:10:53.666 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:10:53.667 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:53.667 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:10:53.667 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:10:53.667 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:53.667 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:53.668 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:53.668 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:53.668 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:10:53.668 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:10:53.668 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:10:53.669 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:10:53.669 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:10:53.669 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:10:53.669 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:53.669 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:10:53.670 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:10:53.670 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:10:53.670 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:10:53.671 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:10:53.671 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:10:53.671 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:10:53.671 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:10:53.671 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:10:53.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-08-24T01:10:53.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:10:53.673 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:10:54.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:10:54.031 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:10:54.031 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:10:54.031 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:10:54.031 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:10:54.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:10:54.032 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:54.032 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:10:54.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836533 2020-08-24T01:10:54.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836533 2020-08-24T01:10:54.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836533' 2020-08-24T01:10:54.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:54.218 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:10:54.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705715 2020-08-24T01:10:54.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705715 2020-08-24T01:10:54.382 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836533 1-38654705715' 2020-08-24T01:10:54.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:54.383 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:10:54.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542194 2020-08-24T01:10:54.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542194 2020-08-24T01:10:54.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836533 1-38654705715 2-60129542194' 2020-08-24T01:10:54.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:54.539 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:10:54.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378673 2020-08-24T01:10:54.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378673 2020-08-24T01:10:54.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836533 1-38654705715 2-60129542194 3-81604378673' 2020-08-24T01:10:54.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:10:54.697 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:10:54.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247856 2020-08-24T01:10:54.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247856 2020-08-24T01:10:54.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836533 1-38654705715 2-60129542194 3-81604378673 4-98784247856' 2020-08-24T01:10:54.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:54.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836533 2020-08-24T01:10:54.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:54.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:10:54.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836533 2020-08-24T01:10:54.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:54.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836533 2020-08-24T01:10:54.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836533' 2020-08-24T01:10:54.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:54.871 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836533 2020-08-24T01:10:55.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836531 -lt 21474836533 2020-08-24T01:10:55.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:10:56.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:10:56.241 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:10:56.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836533 -lt 21474836533 2020-08-24T01:10:56.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:56.620 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705715 2020-08-24T01:10:56.621 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:56.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:10:56.621 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705715 2020-08-24T01:10:56.621 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:56.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705715 2020-08-24T01:10:56.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705715' 2020-08-24T01:10:56.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:10:56.623 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705715 2020-08-24T01:10:56.972 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705716 -lt 38654705715 2020-08-24T01:10:56.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:56.974 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542194 2020-08-24T01:10:56.974 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:56.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:10:56.976 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:56.977 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542194 2020-08-24T01:10:56.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542194 2020-08-24T01:10:56.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542194' 2020-08-24T01:10:56.978 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:10:56.978 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542194 2020-08-24T01:10:57.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542194 -lt 60129542194 2020-08-24T01:10:57.336 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:57.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378673 2020-08-24T01:10:57.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:57.338 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:10:57.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378673 2020-08-24T01:10:57.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:57.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378673 2020-08-24T01:10:57.342 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378673' 2020-08-24T01:10:57.342 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378673 2020-08-24T01:10:57.342 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:10:57.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378673 -lt 81604378673 2020-08-24T01:10:57.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:10:57.729 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247856 2020-08-24T01:10:57.730 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:10:57.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:10:57.730 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247856 2020-08-24T01:10:57.730 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:10:57.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247856 2020-08-24T01:10:57.731 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247856' 2020-08-24T01:10:57.731 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:10:57.731 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247856 2020-08-24T01:10:58.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247856 -lt 98784247856 2020-08-24T01:11:01.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-08-24T01:11:01.117 INFO:tasks.workunit.client.0.smithi060.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-08-24T01:11:01.118 INFO:tasks.workunit.client.0.smithi060.stderr: 2020-08-24T01:11:01.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-08-24T01:11:01.118 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:11:01.118 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:11:01.119 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:11:01.119 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:11:01.119 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:11:01.119 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:11:01.119 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:11:01.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:11:01.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:01.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:11:01.120 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:11:01.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-08-24T01:11:01.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-08-24T01:11:01.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-08-24T01:11:01.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:11:01.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-08-24T01:11:01.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-08-24T01:11:01.134 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-08-24T01:11:01.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-08-24T01:11:01.134 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-08-24T01:11:01.135 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-08-24T01:11:01.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-08-24T01:11:01.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-08-24T01:11:01.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-08-24T01:11:01.136 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:01.136 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:01.136 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:01.136 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:01.149 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:01.150 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:01.150 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:01.151 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:01.151 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:01.151 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:01.151 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:01.152 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:01.152 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:01.152 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:01.152 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:01.153 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:01.153 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:01.153 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:01.153 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:01.169 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:01.169 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:01.170 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:01.170 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:01.170 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:01.170 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:01.170 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:01.171 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:01.171 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:01.171 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:01.171 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:01.171 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:01.171 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:01.172 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:01.172 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:01.172 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:01.172 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:01.172 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:01.173 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:01.173 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:01.174 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:01.174 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:01.318 INFO:tasks.workunit.client.0.smithi060.stderr:noout is unset 2020-08-24T01:11:01.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-08-24T01:11:02.188 INFO:tasks.workunit.client.0.smithi060.stderr:nobackfill is unset 2020-08-24T01:11:02.203 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:11:02.204 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:11:02.204 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:11:02.204 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:11:02.205 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:11:02.205 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:11:02.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:11:02.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations' 2020-08-24T01:11:02.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations' 2020-08-24T01:11:02.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:11:02.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:11:02.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:11:02.227 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:11:02.227 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:11:02.234 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:11:02.234 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:11:02.234 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:11:02.234 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:11:02.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:11:02.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:11:02.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:11:02.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:11:02.416 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:11:02.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:11:02.785 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:11:02.786 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:11:02.786 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:11:02.786 INFO:tasks.workunit.client.0.smithi060.stderr:4' 2020-08-24T01:11:02.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:11:02.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:11:02.787 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:11:02.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836537 2020-08-24T01:11:02.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836537 2020-08-24T01:11:02.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836537' 2020-08-24T01:11:02.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:11:02.967 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:11:03.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705719 2020-08-24T01:11:03.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705719 2020-08-24T01:11:03.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836537 1-38654705719' 2020-08-24T01:11:03.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:11:03.134 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:11:03.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542198 2020-08-24T01:11:03.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542198 2020-08-24T01:11:03.297 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836537 1-38654705719 2-60129542198' 2020-08-24T01:11:03.298 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:11:03.298 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:11:03.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378677 2020-08-24T01:11:03.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378677 2020-08-24T01:11:03.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836537 1-38654705719 2-60129542198 3-81604378677' 2020-08-24T01:11:03.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:11:03.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:11:03.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=98784247859 2020-08-24T01:11:03.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 98784247859 2020-08-24T01:11:03.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836537 1-38654705719 2-60129542198 3-81604378677 4-98784247859' 2020-08-24T01:11:03.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:11:03.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836537 2020-08-24T01:11:03.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:11:03.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:11:03.624 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836537 2020-08-24T01:11:03.624 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:11:03.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836537 2020-08-24T01:11:03.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836537' 2020-08-24T01:11:03.627 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:11:03.627 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836537 2020-08-24T01:11:04.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 0 -lt 21474836537 2020-08-24T01:11:04.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:11:05.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:11:05.031 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:11:05.448 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836537 -lt 21474836537 2020-08-24T01:11:05.448 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:11:05.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705719 2020-08-24T01:11:05.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:11:05.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:11:05.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705719 2020-08-24T01:11:05.452 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:11:05.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705719 2020-08-24T01:11:05.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705719' 2020-08-24T01:11:05.454 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705719 2020-08-24T01:11:05.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:11:05.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705719 -lt 38654705719 2020-08-24T01:11:05.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:11:05.935 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542198 2020-08-24T01:11:05.935 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:11:05.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:11:05.938 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542198 2020-08-24T01:11:05.938 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:11:05.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542198 2020-08-24T01:11:05.943 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542198' 2020-08-24T01:11:05.943 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:11:05.950 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542198 2020-08-24T01:11:06.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542198 -lt 60129542198 2020-08-24T01:11:06.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:11:06.327 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378677 2020-08-24T01:11:06.327 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:11:06.328 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:11:06.329 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378677 2020-08-24T01:11:06.329 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:11:06.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378677 2020-08-24T01:11:06.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378677' 2020-08-24T01:11:06.334 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378677 2020-08-24T01:11:06.335 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:11:06.755 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378677 -lt 81604378677 2020-08-24T01:11:06.755 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:11:06.756 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-98784247859 2020-08-24T01:11:06.756 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:11:06.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:11:06.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-98784247859 2020-08-24T01:11:06.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:11:06.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=98784247859 2020-08-24T01:11:06.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 98784247859' 2020-08-24T01:11:06.782 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:11:06.782 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 98784247859 2020-08-24T01:11:07.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 98784247860 -lt 98784247859 2020-08-24T01:11:07.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:11:07.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:07.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:07.815 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-08-24T01:11:07.815 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:07.816 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:07.816 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:07.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:07.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:07.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:07.818 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:08.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:08.236 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:08.236 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:08.236 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:08.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:08.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T01:11:08.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:11:08.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T01:11:08.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:08.689 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:08.689 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:08.785 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:08.786 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:08.786 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:08.786 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:08.786 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:08.787 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:08.787 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:08.787 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:08.787 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:08.787 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:08.788 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:08.788 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:08.788 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:08.788 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:08.788 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:08.789 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:08.789 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:08.789 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:08.789 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:08.789 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:08.791 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:08.791 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:08.791 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:08.791 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:08.791 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:08.792 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:08.792 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:08.792 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:08.792 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:08.793 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:08.793 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:08.793 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:08.794 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:08.794 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:08.794 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:08.794 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:08.794 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:08.795 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:08.795 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:08.795 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:08.795 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:08.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:08.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:08.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:08.904 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:08.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:08.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:08.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:08.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:08.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:09.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:09.256 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:09.256 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:09.256 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:09.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:09.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:09.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:09.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:09.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:09.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:09.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:09.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:09.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:10.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=16806866 2020-08-24T01:11:10.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 16806866 '!=' null 2020-08-24T01:11:10.176 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:10.176 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:10.176 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:10.176 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:10.267 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:10.267 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:10.267 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:10.268 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:10.268 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:10.268 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:10.268 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:10.269 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:10.269 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:10.269 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:10.269 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:10.269 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:10.271 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:10.271 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:10.271 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:10.272 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:10.272 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:10.272 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:10.272 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:10.273 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:10.273 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:10.273 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:10.274 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:10.274 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:10.274 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:10.274 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:10.274 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:10.275 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:10.275 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:10.275 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:10.275 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:10.275 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:10.276 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:10.276 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:10.276 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:10.276 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:10.277 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:10.277 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:10.277 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:10.277 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:10.277 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:10.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:10.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:10.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:10.377 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:10.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:10.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:10.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:10.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:10.379 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:10.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:10.741 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:10.741 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:10.741 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:11.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:11.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:11.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:11.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:11.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:11.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:11.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:11.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:11.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:11.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=16806866 2020-08-24T01:11:11.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 16806866 '!=' null 2020-08-24T01:11:11.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:11.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:11.661 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:11.661 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:11.743 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:11.744 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:11.744 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:11.748 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:11.748 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:11.748 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:11.749 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:11.749 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:11.749 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:11.749 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:11.750 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:11.750 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:11.750 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:11.751 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:11.751 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:11.751 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:11.752 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:11.752 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:11.752 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:11.752 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:11.752 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:11.753 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:11.753 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:11.753 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:11.754 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:11.754 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:11.754 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:11.754 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:11.755 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:11.755 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:11.755 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:11.756 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:11.756 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:11.757 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:11.757 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:11.757 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:11.758 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:11.759 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:11.759 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:11.759 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:11.760 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:11.761 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:11.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:11.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:11.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:11.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:11.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:11.857 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:11.857 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:11.858 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:12.230 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:12.230 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:12.231 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:12.231 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:12.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:12.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:12.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:12.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:12.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:12.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:12.698 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:12.698 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:12.698 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:13.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13627528 2020-08-24T01:11:13.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13627528 '!=' null 2020-08-24T01:11:13.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:13.172 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:13.172 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:13.172 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:13.264 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:13.264 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:13.264 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:13.265 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:13.265 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:13.265 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:13.265 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:13.265 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:13.266 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:13.266 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:13.266 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:13.266 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:13.266 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:13.267 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:13.269 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:13.269 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:13.269 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:13.269 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:13.270 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:13.271 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:13.271 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:13.272 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:13.272 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:13.272 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:13.272 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:13.272 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:13.273 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:13.273 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:13.273 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:13.273 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:13.274 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:13.274 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:13.274 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:13.275 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:13.275 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:13.275 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:13.275 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:13.275 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:13.275 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:13.276 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:13.276 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:13.276 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:13.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:13.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:13.375 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:13.375 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:13.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:13.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:13.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:13.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:13.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:13.720 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:13.721 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:13.721 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:14.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:14.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:14.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:14.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:14.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:14.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:14.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:14.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:14.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:14.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11632606 2020-08-24T01:11:14.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11632606 '!=' null 2020-08-24T01:11:14.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:14.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:14.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:14.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:14.714 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:14.714 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:14.714 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:14.714 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:14.715 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:14.715 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:14.715 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:14.715 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:14.715 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:14.716 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:14.716 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:14.716 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:14.716 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:14.716 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:14.717 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:14.717 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:14.717 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:14.717 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:14.717 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:14.719 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:14.719 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:14.720 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:14.720 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:14.720 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:14.720 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:14.720 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:14.721 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:14.721 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:14.721 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:14.721 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:14.721 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:14.722 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:14.722 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:14.722 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:14.722 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:14.722 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:14.723 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:14.723 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:14.724 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:14.724 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:14.724 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:14.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:14.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:14.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:14.835 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:14.836 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:14.836 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:14.837 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:14.837 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:14.837 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:15.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:15.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:15.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:15.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:15.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:15.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:15.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:15.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:15.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:15.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:15.647 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:15.647 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:15.647 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:16.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20964656 2020-08-24T01:11:16.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20964656 '!=' null 2020-08-24T01:11:16.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:16.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:16.178 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:16.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:16.262 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:16.262 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:16.262 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:16.263 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:16.263 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:16.263 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:16.263 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:16.263 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:16.264 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:16.264 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:16.264 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:16.264 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:16.264 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:16.265 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:16.265 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:16.265 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:16.265 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:16.265 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:16.266 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:16.266 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:16.266 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:16.266 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:16.266 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:16.266 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:16.267 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:16.267 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:16.267 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:16.267 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:16.267 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:16.268 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:16.268 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:16.268 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:16.268 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:16.268 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:16.268 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:16.269 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:16.269 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:16.269 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:16.269 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:16.269 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:16.270 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:16.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:16.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:16.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:16.372 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:16.373 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:16.373 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:16.373 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:16.373 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:16.374 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:16.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:16.746 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:16.746 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:16.747 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:17.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:17.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:17.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:17.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:17.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:17.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:17.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:17.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:17.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:17.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20964656 2020-08-24T01:11:17.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20964656 '!=' null 2020-08-24T01:11:17.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:17.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:17.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:17.736 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:17.818 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:17.819 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:17.819 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:17.819 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:17.819 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:17.819 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:17.820 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:17.820 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:17.820 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:17.820 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:17.820 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:17.822 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:17.822 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:17.823 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:17.824 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:17.825 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:17.826 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:17.826 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:17.827 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:17.827 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:17.829 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:17.832 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:17.832 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:17.832 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:17.832 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:17.832 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:17.833 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:17.833 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:17.833 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:17.833 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:17.833 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:17.834 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:17.834 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:17.834 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:17.834 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:17.834 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:17.834 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:17.835 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:17.835 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:17.835 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:17.835 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:17.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:17.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:17.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:17.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:17.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:17.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:17.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:17.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:17.933 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:18.286 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:18.287 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:18.287 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:18.287 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:18.820 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:18.820 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:18.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:18.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:18.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:18.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:18.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:18.822 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:18.822 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:19.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18343721 2020-08-24T01:11:19.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18343721 '!=' null 2020-08-24T01:11:19.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:19.259 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:19.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:19.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:19.352 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:19.352 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:19.353 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:19.354 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:19.355 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:19.356 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:19.357 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:19.358 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:19.359 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:19.360 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:19.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:19.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:19.465 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:19.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:19.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:19.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:19.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:19.466 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:19.467 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:19.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:19.815 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:19.815 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:19.815 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:20.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:20.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:20.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:20.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:20.259 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:20.259 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:20.259 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:20.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:20.259 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:20.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27079038 2020-08-24T01:11:20.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27079038 '!=' null 2020-08-24T01:11:20.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:20.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:20.729 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:20.729 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:20.818 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:20.818 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:20.819 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:20.819 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:20.819 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:20.819 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:20.819 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:20.820 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:20.820 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:20.820 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:20.820 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:20.820 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:20.821 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:20.821 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:20.821 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:20.821 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:20.821 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:20.821 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:20.822 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:20.822 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:20.822 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:20.822 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:20.822 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:20.823 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:20.823 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:20.823 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:20.824 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:20.824 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:20.824 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:20.824 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:20.825 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:20.825 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:20.825 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:20.835 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:20.835 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:20.837 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:20.837 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:20.837 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:20.837 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:20.837 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:20.838 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:20.839 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:20.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:20.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:20.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:20.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:20.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:20.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:20.931 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:20.932 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:21.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:21.294 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:21.294 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:21.294 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:21.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:21.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:21.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:21.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:21.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:21.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:21.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:21.765 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:21.765 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:22.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18343987 2020-08-24T01:11:22.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18343987 '!=' null 2020-08-24T01:11:22.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:22.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:22.247 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:22.247 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:22.336 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:22.336 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:22.337 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:22.337 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:22.337 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:22.337 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:22.337 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:22.338 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:22.338 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:22.338 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:22.338 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:22.338 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:22.339 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:22.339 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:22.339 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:22.339 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:22.339 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:22.340 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:22.340 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:22.340 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:22.340 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:22.340 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:22.340 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:22.341 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:22.341 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:22.341 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:22.341 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:22.342 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:22.342 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:22.343 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:22.343 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:22.343 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:22.343 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:22.343 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:22.344 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:22.344 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:22.344 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:22.344 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:22.344 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:22.344 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:22.345 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:22.345 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:22.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:22.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:22.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:22.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:22.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:22.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:22.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:22.448 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:22.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:22.795 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:22.795 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:22.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:23.259 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:23.259 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:23.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:23.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:23.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:23.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:23.261 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:23.261 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:23.261 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:23.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18343987 2020-08-24T01:11:23.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18343987 '!=' null 2020-08-24T01:11:23.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:23.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:23.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:23.737 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:23.824 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:23.825 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:23.825 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:23.825 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:23.825 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:23.826 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:23.826 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:23.827 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:23.827 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:23.827 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:23.827 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:23.827 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:23.828 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:23.828 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:23.828 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:23.828 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:23.828 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:23.828 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:23.829 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:23.829 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:23.829 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:23.829 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:23.830 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:23.830 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:23.830 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:23.830 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:23.830 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:23.830 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:23.831 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:23.831 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:23.831 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:23.831 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:23.831 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:23.832 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:23.832 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:23.832 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:23.832 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:23.832 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:23.832 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:23.833 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:23.833 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:23.833 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:23.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:23.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:23.935 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:23.935 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:23.936 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:23.936 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:23.936 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:23.936 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:24.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:24.311 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:24.311 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:24.312 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:24.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:24.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:24.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:24.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:24.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:24.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:24.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:24.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:24.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:25.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18343934 2020-08-24T01:11:25.261 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18343934 '!=' null 2020-08-24T01:11:25.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:25.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:25.264 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:25.265 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:25.349 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:25.349 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:25.350 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:25.350 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:25.351 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:25.351 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:25.351 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:25.351 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:25.352 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:25.352 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:25.352 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:25.352 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:25.352 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:25.353 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:25.353 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:25.353 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:25.353 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 151, 2020-08-24T01:11:25.354 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:25.354 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:25.354 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:25.354 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:25.354 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:25.355 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:25.355 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:25.355 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:25.355 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:25.355 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:25.359 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:25.359 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "7.0", 2020-08-24T01:11:25.359 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 254, 2020-08-24T01:11:25.359 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:25.360 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:25.360 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:25.360 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:25.361 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:25.361 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:25.361 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:25.361 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:25.362 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:25.362 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:25.362 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:25.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:25.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:25.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:25.469 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:25.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:25.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:25.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:25.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:25.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:25.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:25.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:25.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:25.806 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:26.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:26.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:26.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:26.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:26.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:26.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:26.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:26.268 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:26.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:26.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27952744 2020-08-24T01:11:26.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27952744 '!=' null 2020-08-24T01:11:26.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:26.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:26.882 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:26.882 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:26.967 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:26.967 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:26.967 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:26.967 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:26.968 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:26.968 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:26.968 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:26.968 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:26.968 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:26.969 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:26.969 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:26.969 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:26.969 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:26.969 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:26.970 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:26.970 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:26.970 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:26.970 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:26.970 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:26.975 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:26.975 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:26.975 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:26.976 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:26.976 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:26.976 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:26.977 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:26.977 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:26.977 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:26.977 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:26.977 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:26.978 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:26.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:27.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:27.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:27.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:27.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:27.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:27.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:27.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:27.078 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:27.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:27.439 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:27.439 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:27.439 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:27.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:27.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:27.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:27.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:27.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:27.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:27.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:27.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:27.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:28.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14413396 2020-08-24T01:11:28.478 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14413396 '!=' null 2020-08-24T01:11:28.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:28.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:28.479 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:28.479 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:28.564 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:28.565 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:28.565 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:28.565 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:28.566 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:28.566 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:28.566 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:28.566 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:28.567 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:28.567 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:28.567 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:28.567 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:28.567 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:28.568 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:28.568 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:28.568 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:28.568 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:28.569 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:28.569 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:28.569 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:28.570 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:28.570 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:28.570 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:28.570 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:28.571 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:28.571 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:28.571 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:28.571 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:28.572 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:28.572 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:28.572 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:28.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:28.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:28.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:28.677 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:28.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:28.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:28.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:28.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:28.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:29.038 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:11:29.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:29.038 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:29.039 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:29.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-08-24T01:11:29.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:11:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:29.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:29.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:29.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:29.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:29.991 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14413396 2020-08-24T01:11:29.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14413396 '!=' null 2020-08-24T01:11:29.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:29.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:29.993 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:29.993 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:30.080 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:30.080 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:30.080 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:30.080 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:30.081 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:30.081 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:30.081 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:30.081 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:30.081 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:30.081 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:30.082 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:30.082 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:30.082 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:30.082 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:30.082 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:30.083 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:30.083 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:30.083 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:30.083 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:30.083 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:30.084 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:30.084 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:30.084 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:30.084 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:30.084 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:30.084 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:30.085 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:30.085 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:30.085 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:30.085 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:30.085 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:30.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:30.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:30.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:30.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:30.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:30.200 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:30.200 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:30.200 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:30.200 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:30.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:30.544 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:30.544 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:30.545 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:31.003 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:31.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-08-24T01:11:31.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:11:31.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-08-24T01:11:31.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:31.005 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:31.005 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:31.096 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:31.097 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:31.098 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:31.098 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:31.099 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:31.099 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:31.100 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:31.100 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:31.101 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:31.101 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:31.102 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:31.103 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:31.103 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:31.104 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:31.104 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:31.104 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:31.107 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:31.108 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:31.108 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:31.108 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:31.109 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:31.109 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:31.109 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:31.109 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:31.109 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:31.110 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:31.110 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:31.110 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:31.110 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:31.111 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:31.111 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:31.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:31.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:31.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:31.208 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:31.208 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:31.208 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:31.209 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:31.210 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:31.210 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:31.568 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:31.569 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:31.569 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:31.570 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:32.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:32.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:32.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:32.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:32.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:32.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:32.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:32.031 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:32.031 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:32.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26206369 2020-08-24T01:11:32.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26206369 '!=' null 2020-08-24T01:11:32.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:32.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:32.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:32.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:32.643 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:32.643 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:32.644 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:32.644 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:32.644 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:32.644 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:32.645 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:32.645 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:32.645 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:32.645 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:32.646 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:32.646 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:32.646 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:32.647 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:32.647 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:32.647 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:32.647 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:32.648 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:32.648 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:32.648 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:32.648 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:32.648 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:32.649 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:32.649 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:32.649 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:32.649 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:32.649 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:32.650 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:32.650 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:32.650 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:32.650 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:32.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:32.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:32.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:32.755 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:32.755 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:32.756 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:32.756 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:32.756 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:32.756 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:33.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:33.115 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:33.115 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:33.115 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:33.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:33.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:33.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:33.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:33.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:33.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:33.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:33.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:33.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:34.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26206369 2020-08-24T01:11:34.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26206369 '!=' null 2020-08-24T01:11:34.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:34.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:34.061 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:34.061 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:34.162 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:34.163 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:34.163 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:34.163 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:34.163 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:34.163 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:34.164 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:34.164 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:34.164 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:34.164 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:34.164 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:34.165 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:34.165 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:34.165 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:34.165 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:34.165 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:34.165 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:34.166 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:34.166 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:34.166 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:34.166 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:34.166 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:34.180 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:34.180 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:34.180 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:34.181 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:34.181 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:34.181 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:34.181 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:34.181 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:34.182 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:34.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:34.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:34.271 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:34.272 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:34.272 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:34.272 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:34.273 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:34.273 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:34.273 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:34.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:34.663 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:34.664 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:34.664 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:35.139 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:35.139 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:35.139 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:35.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:35.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:35.140 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:35.141 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:35.141 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:35.141 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:35.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11792724 2020-08-24T01:11:35.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11792724 '!=' null 2020-08-24T01:11:35.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:35.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:35.598 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:35.598 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:35.687 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:35.688 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:35.688 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:35.688 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:35.689 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:35.689 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:35.689 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:35.689 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:35.690 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:35.690 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:35.690 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:35.691 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:35.691 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:35.691 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:35.693 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:35.693 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:35.693 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:35.694 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:35.694 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:35.694 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:35.695 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:35.695 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:35.695 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:35.696 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:35.696 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:35.696 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:35.696 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:35.697 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:35.697 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:35.697 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:35.697 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:35.710 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:35.796 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:35.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:35.798 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:35.798 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:35.798 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:35.798 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:35.799 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:35.799 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:36.150 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:36.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:36.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:36.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:36.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:36.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:36.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:36.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:36.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:36.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:36.656 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:36.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:36.657 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:37.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=22390779 2020-08-24T01:11:37.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 22390779 '!=' null 2020-08-24T01:11:37.104 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:37.104 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:37.104 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:37.104 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:37.184 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:37.185 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:37.185 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:37.185 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:37.187 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:37.187 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:37.188 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:37.188 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:37.188 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:37.188 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:37.189 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:37.189 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:37.194 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:37.194 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:37.194 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:37.195 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:37.195 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:37.195 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:37.195 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:37.196 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:37.196 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:37.196 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:37.196 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:37.196 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:37.197 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:37.197 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:37.199 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:37.199 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:37.199 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:37.199 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:37.200 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:37.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:37.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:37.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:37.296 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:37.296 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:37.296 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:37.296 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:37.297 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:37.297 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:37.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:37.653 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:37.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:37.654 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:38.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:38.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:38.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:38.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:38.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:38.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:38.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:38.128 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:38.128 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:38.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19256330 2020-08-24T01:11:38.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19256330 '!=' null 2020-08-24T01:11:38.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:38.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:38.619 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:38.619 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:38.709 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:38.709 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:38.710 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:38.710 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:38.710 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:38.710 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:38.710 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:38.721 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:38.722 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:38.722 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:38.722 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:38.722 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:38.723 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:38.723 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:38.723 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:38.723 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:38.723 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:38.724 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:38.724 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:38.724 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:38.724 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:38.724 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:38.725 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:38.725 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:38.725 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:38.725 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:38.725 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:38.726 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:38.726 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:38.726 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:38.726 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:38.727 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:38.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:38.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:38.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:38.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:38.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:38.818 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:38.818 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:38.819 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:39.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:39.180 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:39.181 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:39.181 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:39.674 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:39.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:39.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:39.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:39.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:39.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:39.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:39.676 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:39.676 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:40.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26205115 2020-08-24T01:11:40.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26205115 '!=' null 2020-08-24T01:11:40.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:40.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:40.174 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:40.174 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:40.263 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:40.263 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:40.263 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:40.264 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:40.264 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:40.264 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:40.265 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:40.265 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:40.265 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:40.265 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:40.265 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:40.266 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:40.266 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:40.266 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:40.266 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:40.266 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:40.267 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:40.267 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:40.267 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:40.267 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:40.267 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:40.268 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:40.268 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:40.268 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:40.268 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:40.268 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:40.268 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:40.269 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:40.269 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:40.269 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:40.269 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:40.271 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:40.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:40.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:40.374 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:40.374 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:40.374 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:40.375 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:40.375 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:40.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:40.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:40.788 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:40.788 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:40.789 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:41.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:41.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:41.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:41.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:41.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:41.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:41.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:41.240 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:41.240 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:41.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26205115 2020-08-24T01:11:41.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26205115 '!=' null 2020-08-24T01:11:41.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:41.672 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:41.673 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:41.673 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:41.766 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:41.767 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:41.767 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:41.767 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:41.767 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:41.767 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:41.768 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:41.768 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:41.768 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:41.768 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:41.768 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:41.769 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:41.769 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:41.769 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:41.769 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:41.769 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:41.772 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:41.772 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:41.772 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:41.773 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:41.773 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:41.773 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:41.773 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:41.773 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:41.773 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:41.774 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:41.774 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:41.774 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:41.774 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:41.774 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:41.775 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:41.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:41.876 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:41.876 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:41.876 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:41.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:41.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:41.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:41.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:41.878 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:42.230 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:42.232 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:42.232 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:42.232 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:42.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:42.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:42.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:42.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:42.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:42.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:42.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:42.704 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:42.704 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:43.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18343564 2020-08-24T01:11:43.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18343564 '!=' null 2020-08-24T01:11:43.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:43.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:43.160 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:43.160 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:43.238 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:43.239 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:43.239 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:43.239 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:43.240 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:43.240 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:43.240 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:43.240 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:43.240 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:43.241 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:43.241 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:43.241 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:43.241 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:43.241 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:43.241 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:43.242 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:43.242 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:43.242 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:43.242 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:43.242 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:43.243 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:43.243 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:43.243 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:43.243 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:43.243 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:43.243 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:43.244 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:43.244 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:43.244 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:43.244 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:43.244 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:43.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:43.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:43.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:43.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:43.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:43.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:43.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:43.350 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:43.351 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:43.692 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:43.693 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:43.693 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:43.693 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:44.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:44.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:44.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:44.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:44.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:44.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:44.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:44.196 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:44.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:44.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18343384 2020-08-24T01:11:44.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18343384 '!=' null 2020-08-24T01:11:44.800 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:44.800 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:44.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:44.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:44.892 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:44.892 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:44.892 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:44.893 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:44.893 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:44.893 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:44.893 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:44.893 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:44.894 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:44.894 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:44.894 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:44.894 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:44.894 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:44.895 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:44.895 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:44.895 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:44.895 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:44.895 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:44.895 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:44.896 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:44.896 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:44.897 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:44.897 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:44.897 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:44.898 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:44.898 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:44.898 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:44.898 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:44.898 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:44.899 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:44.899 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:44.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:45.002 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:45.003 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:45.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:45.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:45.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:45.003 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:45.004 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:45.004 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:45.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:45.357 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:45.358 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:45.358 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:45.991 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:45.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:45.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:45.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:45.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:45.993 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:45.994 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:45.995 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:45.995 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:46.479 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27951996 2020-08-24T01:11:46.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27951996 '!=' null 2020-08-24T01:11:46.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:46.480 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:46.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:46.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:46.569 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:46.569 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:46.569 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:46.570 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:46.570 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:46.570 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:46.570 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:46.570 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:46.571 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:46.571 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:46.571 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:46.571 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:46.571 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:46.572 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:46.572 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:46.572 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:46.572 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:46.572 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:46.573 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:46.573 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:46.573 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:46.574 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:46.574 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:46.574 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:46.574 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:46.575 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:46.575 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:46.575 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:46.575 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:46.575 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:46.575 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:46.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:46.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:46.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:46.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:46.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:46.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:46.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:46.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:46.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:47.039 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:47.040 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:47.040 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:47.041 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:47.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:47.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:47.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:47.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:47.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:47.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:47.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:47.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:47.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:48.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19216980 2020-08-24T01:11:48.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19216980 '!=' null 2020-08-24T01:11:48.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:48.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:48.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:48.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:48.199 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:48.200 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:48.200 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:48.200 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:48.200 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:48.201 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:48.201 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:48.201 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:48.201 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:48.202 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:48.202 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:48.202 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:48.202 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:48.202 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:48.203 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:48.203 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:48.203 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:48.203 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:48.204 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:48.204 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:48.204 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:48.204 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:48.204 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:48.205 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:48.205 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:48.205 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:48.205 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:48.205 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:48.206 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:48.206 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:48.206 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:48.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:48.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:48.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:48.310 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:48.310 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:48.310 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:48.311 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:48.311 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:48.311 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:48.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:48.702 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:48.702 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:48.702 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:49.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:49.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:49.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:49.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:49.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:49.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:49.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:49.161 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:49.162 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:49.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19216980 2020-08-24T01:11:49.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19216980 '!=' null 2020-08-24T01:11:49.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:49.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:49.668 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:49.669 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:49.759 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:49.759 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:49.760 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:49.760 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:49.760 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [ 2020-08-24T01:11:49.761 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:49.761 INFO:tasks.workunit.client.0.smithi060.stdout: "priority": 110, 2020-08-24T01:11:49.761 INFO:tasks.workunit.client.0.smithi060.stdout: "items": [ 2020-08-24T01:11:49.761 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:49.762 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:49.762 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:49.763 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:49.763 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:49.763 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:49.763 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:49.763 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:11:49.764 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:49.764 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:49.764 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "3.0", 2020-08-24T01:11:49.764 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 151, 2020-08-24T01:11:49.764 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:49.765 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:49.765 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:49.765 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:49.765 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:49.766 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:49.766 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:49.766 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:49.766 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:49.767 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:49.767 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:49.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:49.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:49.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:49.869 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:49.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:49.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:49.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:49.871 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:49.872 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:50.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:50.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:50.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:50.219 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:50.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:50.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:50.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:50.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:50.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:50.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:50.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:50.669 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:50.669 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:51.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27952834 2020-08-24T01:11:51.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27952834 '!=' null 2020-08-24T01:11:51.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:51.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:51.107 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:51.107 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:51.184 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:51.185 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:51.187 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:51.187 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:51.187 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:51.188 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:51.189 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:51.189 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:51.189 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:51.189 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:51.190 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:51.190 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:51.191 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:51.192 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:51.192 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:51.192 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:51.192 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:51.193 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:51.193 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:51.193 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:51.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:51.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:51.293 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:51.294 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:51.295 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:51.295 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:51.295 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:51.295 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:51.296 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:51.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:51.642 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:51.643 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:51.644 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:52.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:52.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:52.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:52.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:52.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:52.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:52.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:52.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:52.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:52.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27516573 2020-08-24T01:11:52.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27516573 '!=' null 2020-08-24T01:11:52.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:52.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:52.643 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:52.643 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:52.737 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:52.737 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:52.737 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:52.738 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:52.738 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:52.738 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:52.738 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:52.738 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:52.739 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:52.739 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:52.739 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:52.739 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:52.739 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:52.740 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:52.740 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:52.740 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:52.740 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:52.740 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:52.740 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:52.741 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:52.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:52.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:52.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:52.851 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:52.851 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:52.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:52.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:52.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:52.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:53.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:53.195 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:53.196 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:53.196 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:53.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:53.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:53.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:53.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:53.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:53.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:53.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:53.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:53.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:54.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13103381 2020-08-24T01:11:54.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13103381 '!=' null 2020-08-24T01:11:54.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:54.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:54.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:54.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:54.296 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:54.296 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:54.297 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:54.297 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:54.297 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:54.298 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:54.298 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:54.298 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:54.299 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:54.299 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:54.299 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:54.300 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:54.300 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:54.300 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:54.301 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:54.301 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:54.303 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:54.303 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:54.303 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:54.304 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:54.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:54.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:54.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:54.406 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:54.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:54.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:54.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:54.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:54.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:54.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:11:54.766 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:54.767 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:54.767 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:55.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-08-24T01:11:55.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-08-24T01:11:55.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:55.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:55.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:55.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:55.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:55.233 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:55.234 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:55.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13103381 2020-08-24T01:11:55.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13103381 '!=' null 2020-08-24T01:11:55.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:55.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:55.668 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:55.668 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:55.757 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:55.757 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:55.758 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:55.758 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:55.758 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:55.758 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:55.758 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:55.759 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:55.759 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:55.759 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:55.759 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:55.759 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:55.760 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:55.760 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:55.760 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:55.760 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:55.760 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:55.761 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:55.761 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:55.761 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:55.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:55.872 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:55.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:55.873 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:55.873 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:55.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:55.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:55.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:55.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:56.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:11:56.235 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:56.240 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:56.240 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:56.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:11:56.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-08-24T01:11:56.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:11:56.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-08-24T01:11:56.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:56.681 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:56.681 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:56.762 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:56.763 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:56.763 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:56.763 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:56.764 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:56.764 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:56.764 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:56.764 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:56.764 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:56.764 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:56.765 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:56.766 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:56.767 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:56.767 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:56.767 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:56.768 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:56.769 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:56.769 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:56.769 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:56.769 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:56.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:56.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:56.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:56.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:56.879 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:56.879 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:56.880 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:56.880 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:56.880 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:57.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:11:57.231 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:57.231 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:57.231 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:57.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:11:57.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:11:57.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:57.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:57.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:57.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:57.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:57.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:57.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:58.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11792877 2020-08-24T01:11:58.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11792877 '!=' null 2020-08-24T01:11:58.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:58.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:58.246 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:58.247 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:58.341 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:58.342 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:58.342 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:58.342 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:58.343 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:58.343 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:58.343 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:58.344 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:58.344 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:58.344 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:58.344 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:58.345 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:58.345 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:58.345 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:58.345 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:58.347 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:58.348 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:58.348 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:58.348 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:58.349 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:58.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:11:58.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:11:58.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:11:58.455 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:11:58.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:11:58.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:11:58.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:11:58.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:11:58.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:11:58.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:11:58.804 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:11:58.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:11:58.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:11:59.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:11:59.263 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:11:59.263 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:11:59.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:11:59.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:11:59.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:11:59.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:11:59.265 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:11:59.265 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:11:59.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11792877 2020-08-24T01:11:59.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11792877 '!=' null 2020-08-24T01:11:59.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:11:59.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:59.872 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:11:59.872 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:11:59.963 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:11:59.963 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:11:59.963 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:11:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:11:59.964 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:11:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:11:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:11:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:11:59.965 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:11:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:11:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:11:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:11:59.966 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:11:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:11:59.967 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:11:59.967 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:11:59.970 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:00.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:00.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:00.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:00.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:00.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:00.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:00.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:00.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:00.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:00.429 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:00.430 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:00.430 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:00.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:00.927 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:00.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:00.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:00.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:00.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:00.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:00.929 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:00.930 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:01.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24540883 2020-08-24T01:12:01.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24540883 '!=' null 2020-08-24T01:12:01.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:01.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:01.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:01.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:01.467 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:01.468 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:01.469 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:01.470 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:01.471 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:01.472 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:01.473 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:01.474 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:01.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:01.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:01.578 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:01.579 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:01.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:01.580 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:01.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:01.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:01.581 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:01.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:01.960 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:01.961 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:01.962 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:02.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:02.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:02.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:02.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:02.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:02.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:02.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:02.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:02.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:02.913 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20708788 2020-08-24T01:12:02.913 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20708788 '!=' null 2020-08-24T01:12:02.913 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:02.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:02.914 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:02.915 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:03.005 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:03.005 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:03.022 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:03.023 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:03.023 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:03.023 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:03.023 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:03.024 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:03.024 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:03.024 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:03.025 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:03.025 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:03.025 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:03.033 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:03.033 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:03.034 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:03.034 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:03.034 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:03.035 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:03.035 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:03.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:03.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:03.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:03.117 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:03.117 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:03.117 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:03.117 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:03.118 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:03.118 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:03.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:03.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:03.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:03.476 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:03.939 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:03.939 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:03.939 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:03.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:03.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:03.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:03.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:03.941 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:03.941 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:04.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17471131 2020-08-24T01:12:04.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17471131 '!=' null 2020-08-24T01:12:04.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:04.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:04.404 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:04.404 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:04.480 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:04.481 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:04.481 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:04.482 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:04.483 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:04.483 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:04.483 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:04.484 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:04.484 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:04.484 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:04.484 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:04.490 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:04.490 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:04.491 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:04.492 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:04.492 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:04.492 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:04.492 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:04.492 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:04.493 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:04.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:04.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:04.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:04.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:04.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:04.595 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:04.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:04.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:04.596 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:04.950 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:04.951 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:04.951 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:04.951 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:05.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:05.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:05.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:05.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:05.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:05.453 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:05.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:05.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:05.455 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:05.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17471131 2020-08-24T01:12:05.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17471131 '!=' null 2020-08-24T01:12:05.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:05.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:05.891 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:05.892 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:05.971 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:05.971 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:05.972 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:05.972 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:05.972 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:05.972 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:05.973 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:05.973 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:05.973 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:05.973 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:05.974 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:05.974 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:05.974 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:05.974 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:05.975 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:05.975 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:05.975 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:05.975 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:05.975 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:05.976 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:05.986 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:06.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:06.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:06.082 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:06.082 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:06.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:06.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:06.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:06.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:06.443 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:06.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:06.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:06.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:06.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:06.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:06.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:06.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:06.897 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:06.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:06.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:06.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:06.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:07.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26207260 2020-08-24T01:12:07.338 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26207260 '!=' null 2020-08-24T01:12:07.338 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:07.339 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:07.339 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:07.340 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:07.438 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:07.439 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:07.439 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:07.439 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:07.439 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:07.439 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:07.440 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:07.440 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:07.440 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:07.440 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:07.440 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:07.440 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:07.441 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:07.441 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:07.442 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:07.442 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:07.442 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:07.442 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:07.443 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:07.443 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:07.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:07.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:07.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:07.547 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:07.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:07.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:07.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:07.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:07.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:07.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:07.897 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:07.897 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:07.898 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:08.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:08.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:08.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:08.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:08.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:08.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:08.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:08.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:08.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:09.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18345573 2020-08-24T01:12:09.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18345573 '!=' null 2020-08-24T01:12:09.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:09.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:09.047 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:09.048 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:09.138 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:09.139 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:09.139 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:09.139 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:09.140 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:09.140 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:09.140 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:09.141 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:09.141 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:09.141 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:09.142 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:09.142 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:09.142 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:09.144 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:09.144 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:09.145 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:09.145 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:09.145 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:09.146 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:09.146 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:09.147 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:09.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:09.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:09.248 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:09.248 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:09.248 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:09.248 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:09.249 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:09.249 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:09.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:09.603 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:09.604 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:09.604 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:10.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:10.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:10.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:10.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:10.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:10.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:10.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:10.122 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:10.122 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:10.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27955437 2020-08-24T01:12:10.577 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27955437 '!=' null 2020-08-24T01:12:10.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:10.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:10.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:10.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:10.666 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:10.666 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:10.667 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:10.667 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:10.667 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:10.667 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:10.667 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:10.668 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:10.668 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:10.668 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:10.668 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:10.668 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:10.669 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:10.669 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:10.669 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:10.669 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:10.669 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:10.669 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:10.670 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:10.670 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:10.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:10.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:10.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:10.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:10.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:10.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:10.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:10.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:10.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:11.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:11.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:11.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:11.122 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:11.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:11.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:11.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:11.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:11.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:11.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:11.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:11.750 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:11.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:12.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18345894 2020-08-24T01:12:12.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18345894 '!=' null 2020-08-24T01:12:12.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:12.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:12.240 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:12.240 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:12.331 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:12.332 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:12.332 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:12.332 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:12.332 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:12.333 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:12.333 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:12.333 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:12.333 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:12.334 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:12.334 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:12.334 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:12.334 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:12.334 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:12.334 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:12.335 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:12.335 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:12.335 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:12.335 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:12.335 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:12.339 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:12.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:12.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:12.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:12.442 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:12.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:12.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:12.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:12.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:12.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:12.857 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:12.858 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:12.858 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:13.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:13.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:13.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:13.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:13.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:13.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:13.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:13.327 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:13.327 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:13.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18345894 2020-08-24T01:12:13.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18345894 '!=' null 2020-08-24T01:12:13.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:13.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:13.812 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:13.812 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:13.901 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:13.902 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:13.902 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:13.902 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:13.902 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:13.902 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:13.903 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:13.903 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:13.903 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:13.903 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:13.903 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:13.904 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:13.904 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:13.904 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:13.904 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:13.904 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:13.904 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:13.905 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:13.906 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:13.906 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:13.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:14.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:14.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:14.011 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:14.012 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:14.012 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:14.012 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:14.012 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:14.012 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:14.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:14.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:14.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:14.371 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:14.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:14.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:14.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:14.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:14.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:14.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:14.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:14.868 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:14.869 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:15.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18345801 2020-08-24T01:12:15.314 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18345801 '!=' null 2020-08-24T01:12:15.314 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:15.314 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:15.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:15.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:15.408 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:15.409 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:15.409 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:15.409 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:15.409 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:15.409 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [ 2020-08-24T01:12:15.410 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:15.410 INFO:tasks.workunit.client.0.smithi060.stdout: "item": "1.0", 2020-08-24T01:12:15.410 INFO:tasks.workunit.client.0.smithi060.stdout: "prio": 110, 2020-08-24T01:12:15.410 INFO:tasks.workunit.client.0.smithi060.stdout: "can_preempt": true 2020-08-24T01:12:15.410 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:15.411 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:15.411 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:15.411 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:15.411 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:15.411 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:15.412 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:15.412 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:15.412 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:15.412 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:15.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:15.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:15.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:15.518 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:15.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:15.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:15.519 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:15.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:15.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:15.872 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:15.873 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:15.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:15.874 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:16.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:16.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:16.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:16.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:16.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:16.386 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:16.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:16.387 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:16.388 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:16.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27081819 2020-08-24T01:12:16.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27081819 '!=' null 2020-08-24T01:12:16.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:16.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:16.864 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:16.864 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:16.954 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:16.954 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:16.957 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:16.957 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:16.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:17.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:17.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:17.065 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:17.066 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:17.066 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:17.066 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:17.066 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:17.067 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:17.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:17.426 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:17.426 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:17.426 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:17.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:17.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:17.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:17.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:17.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:17.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:17.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:17.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:17.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:18.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13103622 2020-08-24T01:12:18.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13103622 '!=' null 2020-08-24T01:12:18.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:18.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:18.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:18.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:18.410 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:18.411 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:18.411 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:18.411 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:18.411 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:18.411 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:18.411 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:18.412 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:18.412 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:18.412 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:18.412 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:18.412 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:18.413 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:18.413 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:18.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:18.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:18.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:18.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:18.520 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:18.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:18.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:18.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:18.527 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:18.876 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-08-24T01:12:18.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:18.877 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:18.878 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:19.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-08-24T01:12:19.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-08-24T01:12:19.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:12:19.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:12:19.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:12:19.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:12:19.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:12:19.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:12:19.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:12:19.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13103622 2020-08-24T01:12:19.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13103622 '!=' null 2020-08-24T01:12:19.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:12:19.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:19.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-08-24T01:12:19.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_recovery_reservations 2020-08-24T01:12:19.857 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:19.857 INFO:tasks.workunit.client.0.smithi060.stdout: "local_reservations": { 2020-08-24T01:12:19.857 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:19.858 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:19.858 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:19.858 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:19.858 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:19.858 INFO:tasks.workunit.client.0.smithi060.stdout: "remote_reservations": { 2020-08-24T01:12:19.859 INFO:tasks.workunit.client.0.smithi060.stdout: "max_allowed": 1, 2020-08-24T01:12:19.859 INFO:tasks.workunit.client.0.smithi060.stdout: "min_priority": 0, 2020-08-24T01:12:19.859 INFO:tasks.workunit.client.0.smithi060.stdout: "queues": [], 2020-08-24T01:12:19.859 INFO:tasks.workunit.client.0.smithi060.stdout: "in_progress": [] 2020-08-24T01:12:19.859 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:19.859 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:19.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:12:19.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:12:19.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:12:19.968 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:12:19.968 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:12:19.969 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:12:19.969 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:12:19.969 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:12:19.970 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:12:20.332 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-08-24T01:12:20.333 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:12:20.333 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:12:20.333 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:12:20.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-08-24T01:12:20.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:12:20.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:12:20.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-08-24T01:12:21.120 INFO:tasks.workunit.client.0.smithi060.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-08-24T01:12:21.121 INFO:tasks.workunit.client.0.smithi060.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-08-24T01:11:27.893146+0000 93'150 118:344 [1,4] 1 [1,4] 1 0'0 2020-08-24T01:08:23.736724+0000 0'0 2020-08-24T01:08:23.736724+0000 0 2020-08-24T01:12:21.121 INFO:tasks.workunit.client.0.smithi060.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-08-24T01:12:16.706983+0000 93'150 119:353 [1,2] 1 [1,2] 1 0'0 2020-08-24T01:08:08.835653+0000 0'0 2020-08-24T01:08:08.835653+0000 0 2020-08-24T01:12:21.122 INFO:tasks.workunit.client.0.smithi060.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-08-24T01:11:52.422102+0000 93'150 118:357 [1,2] 1 [1,2] 1 0'0 2020-08-24T01:08:14.028831+0000 0'0 2020-08-24T01:08:14.028831+0000 0 2020-08-24T01:12:21.122 INFO:tasks.workunit.client.0.smithi060.stdout: 2020-08-24T01:12:21.122 INFO:tasks.workunit.client.0.smithi060.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-08-24T01:12:21.123 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:12:21.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-08-24T01:12:21.132 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-08-24T01:12:21.132 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-08-24T01:12:21.135 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:12:21.136 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:21.137 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:12:21.137 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.50053/ceph-osd.1.asok 2020-08-24T01:12:21.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-08-24T01:12:21.138 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.50053/ceph-osd.1.asok dump_pgstate_history 2020-08-24T01:12:21.223 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:12:21.223 INFO:tasks.workunit.client.0.smithi060.stdout: "pgs": [ 2020-08-24T01:12:21.224 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.224 INFO:tasks.workunit.client.0.smithi060.stdout: "pg": "7.0", 2020-08-24T01:12:21.224 INFO:tasks.workunit.client.0.smithi060.stdout: "currently": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.229 INFO:tasks.workunit.client.0.smithi060.stdout: "history": [ 2020-08-24T01:12:21.229 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.230 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "62", 2020-08-24T01:12:21.230 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.230 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.231 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.231 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.263930+0000", 2020-08-24T01:12:21.231 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:24.263957+0000" 2020-08-24T01:12:21.231 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.231 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.232 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.232 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.263977+0000", 2020-08-24T01:12:21.232 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:24.405866+0000" 2020-08-24T01:12:21.232 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.233 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.233 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.233 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.405867+0000", 2020-08-24T01:12:21.233 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:24.406027+0000" 2020-08-24T01:12:21.238 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.238 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.239 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.239 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.406028+0000", 2020-08-24T01:12:21.240 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:24.406058+0000" 2020-08-24T01:12:21.240 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.241 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.241 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.242 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.406059+0000", 2020-08-24T01:12:21.242 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:24.607268+0000" 2020-08-24T01:12:21.242 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.243 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.243 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.243 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.263966+0000", 2020-08-24T01:12:21.244 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:24.607316+0000" 2020-08-24T01:12:21.244 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.245 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.245 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.246 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.608187+0000", 2020-08-24T01:12:21.246 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.069811+0000" 2020-08-24T01:12:21.246 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.247 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.247 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.247 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.069813+0000", 2020-08-24T01:12:21.248 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.069950+0000" 2020-08-24T01:12:21.248 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.248 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.249 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.249 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.069950+0000", 2020-08-24T01:12:21.249 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.965078+0000" 2020-08-24T01:12:21.250 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.250 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.252 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.252 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.607318+0000", 2020-08-24T01:12:21.252 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.965131+0000" 2020-08-24T01:12:21.253 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.253 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.254 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.257 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.263958+0000", 2020-08-24T01:12:21.257 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.965175+0000" 2020-08-24T01:12:21.258 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.258 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.258 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.259 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:24.263922+0000", 2020-08-24T01:12:21.259 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.965189+0000" 2020-08-24T01:12:21.260 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.260 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.260 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.261 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.263 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "63", 2020-08-24T01:12:21.263 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.264 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.264 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.265 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.965190+0000", 2020-08-24T01:12:21.265 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.965943+0000" 2020-08-24T01:12:21.265 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.266 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.266 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.266 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.267 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "91", 2020-08-24T01:12:21.267 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.267 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.268 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.268 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.965960+0000", 2020-08-24T01:12:21.268 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:25.966000+0000" 2020-08-24T01:12:21.269 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.269 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.269 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.269 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.966025+0000", 2020-08-24T01:12:21.270 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:26.091726+0000" 2020-08-24T01:12:21.270 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.270 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.271 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.271 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:26.091727+0000", 2020-08-24T01:12:21.271 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:26.091835+0000" 2020-08-24T01:12:21.271 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.272 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.272 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.272 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:26.091836+0000", 2020-08-24T01:12:21.272 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:26.091857+0000" 2020-08-24T01:12:21.273 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.274 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.274 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.274 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:26.091858+0000", 2020-08-24T01:12:21.275 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:26.249682+0000" 2020-08-24T01:12:21.275 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.275 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.275 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.276 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.966013+0000", 2020-08-24T01:12:21.276 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:26.249698+0000" 2020-08-24T01:12:21.276 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.277 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.277 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.277 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:26.249836+0000", 2020-08-24T01:12:21.277 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:26.808762+0000" 2020-08-24T01:12:21.278 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.278 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.278 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.279 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:26.808763+0000", 2020-08-24T01:12:21.279 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:26.808901+0000" 2020-08-24T01:12:21.279 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.280 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.280 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.280 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:26.808902+0000", 2020-08-24T01:12:21.280 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.284050+0000" 2020-08-24T01:12:21.281 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.283 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.284 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.284 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:26.249699+0000", 2020-08-24T01:12:21.284 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.284148+0000" 2020-08-24T01:12:21.285 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.285 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.285 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.285 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.966001+0000", 2020-08-24T01:12:21.285 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.284225+0000" 2020-08-24T01:12:21.286 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.286 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.286 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.286 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:25.965946+0000", 2020-08-24T01:12:21.287 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.284250+0000" 2020-08-24T01:12:21.287 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.287 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.287 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.287 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.288 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "91", 2020-08-24T01:12:21.288 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.288 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.288 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.288 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.284252+0000", 2020-08-24T01:12:21.289 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.285089+0000" 2020-08-24T01:12:21.289 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.289 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.289 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.289 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.290 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "98", 2020-08-24T01:12:21.290 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.290 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.290 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.290 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.285103+0000", 2020-08-24T01:12:21.291 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.285143+0000" 2020-08-24T01:12:21.291 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.291 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.291 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.292 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.285171+0000", 2020-08-24T01:12:21.292 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.402664+0000" 2020-08-24T01:12:21.292 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.292 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.292 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.293 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.402666+0000", 2020-08-24T01:12:21.293 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.402920+0000" 2020-08-24T01:12:21.293 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.293 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.293 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.298 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.402922+0000", 2020-08-24T01:12:21.298 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.402986+0000" 2020-08-24T01:12:21.299 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.299 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.299 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.300 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.402987+0000", 2020-08-24T01:12:21.300 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.718249+0000" 2020-08-24T01:12:21.301 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.301 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.301 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.302 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.285156+0000", 2020-08-24T01:12:21.303 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.718273+0000" 2020-08-24T01:12:21.303 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.303 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.303 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.304 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.718482+0000", 2020-08-24T01:12:21.304 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.793799+0000" 2020-08-24T01:12:21.304 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.304 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.304 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.304 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.793800+0000", 2020-08-24T01:12:21.305 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.793921+0000" 2020-08-24T01:12:21.305 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.305 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.305 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.306 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.793922+0000", 2020-08-24T01:12:21.306 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:14.973074+0000" 2020-08-24T01:12:21.306 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.306 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.306 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.307 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.718274+0000", 2020-08-24T01:12:21.307 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:14.973164+0000" 2020-08-24T01:12:21.307 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.307 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.307 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.307 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.285144+0000", 2020-08-24T01:12:21.308 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:14.973287+0000" 2020-08-24T01:12:21.308 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.308 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.308 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.308 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.285090+0000", 2020-08-24T01:12:21.309 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:14.973314+0000" 2020-08-24T01:12:21.309 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.309 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.309 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.309 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.310 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "98", 2020-08-24T01:12:21.310 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.310 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.310 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.310 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:14.973316+0000", 2020-08-24T01:12:21.311 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:14.974641+0000" 2020-08-24T01:12:21.311 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.311 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.311 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.311 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.312 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "101", 2020-08-24T01:12:21.312 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.312 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.312 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.312 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:14.974669+0000", 2020-08-24T01:12:21.312 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:14.974754+0000" 2020-08-24T01:12:21.313 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.313 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.313 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.313 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:14.974807+0000", 2020-08-24T01:12:21.313 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.066522+0000" 2020-08-24T01:12:21.314 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.314 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.314 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.314 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.066523+0000", 2020-08-24T01:12:21.314 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.066660+0000" 2020-08-24T01:12:21.315 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.315 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.315 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.315 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:14.974781+0000", 2020-08-24T01:12:21.315 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.066681+0000" 2020-08-24T01:12:21.315 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.316 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.316 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/WaitActingChange", 2020-08-24T01:12:21.316 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.066682+0000", 2020-08-24T01:12:21.316 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.430360+0000" 2020-08-24T01:12:21.316 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.317 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.317 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.317 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:14.974756+0000", 2020-08-24T01:12:21.317 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.430409+0000" 2020-08-24T01:12:21.317 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.318 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.318 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.318 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:14.974642+0000", 2020-08-24T01:12:21.318 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.430422+0000" 2020-08-24T01:12:21.318 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.318 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.319 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.319 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.319 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "101", 2020-08-24T01:12:21.319 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.319 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.320 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.320 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.430423+0000", 2020-08-24T01:12:21.321 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.431062+0000" 2020-08-24T01:12:21.321 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.321 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.321 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.322 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.322 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "114", 2020-08-24T01:12:21.322 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.322 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.322 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.323 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.431077+0000", 2020-08-24T01:12:21.323 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.431119+0000" 2020-08-24T01:12:21.323 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.323 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.323 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.324 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.431146+0000", 2020-08-24T01:12:21.324 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.557759+0000" 2020-08-24T01:12:21.324 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.324 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.324 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.324 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.557760+0000", 2020-08-24T01:12:21.325 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.557960+0000" 2020-08-24T01:12:21.325 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.325 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.325 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.325 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.557961+0000", 2020-08-24T01:12:21.326 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.558010+0000" 2020-08-24T01:12:21.326 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.326 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.326 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.326 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.558011+0000", 2020-08-24T01:12:21.327 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.692901+0000" 2020-08-24T01:12:21.327 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.327 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.327 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.327 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.431133+0000", 2020-08-24T01:12:21.328 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:15.692935+0000" 2020-08-24T01:12:21.328 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.328 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.328 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.328 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.693637+0000", 2020-08-24T01:12:21.328 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:16.135310+0000" 2020-08-24T01:12:21.329 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.329 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.329 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-08-24T01:12:21.329 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:16.135312+0000", 2020-08-24T01:12:21.329 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:16.135784+0000" 2020-08-24T01:12:21.330 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.330 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.330 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-08-24T01:12:21.330 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:16.135785+0000", 2020-08-24T01:12:21.330 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:16.136711+0000" 2020-08-24T01:12:21.331 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.331 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.331 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Backfilling", 2020-08-24T01:12:21.331 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:16.136712+0000", 2020-08-24T01:12:21.331 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:45.954898+0000" 2020-08-24T01:12:21.331 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.332 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.332 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-08-24T01:12:21.332 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:45.954899+0000", 2020-08-24T01:12:21.332 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:45.955463+0000" 2020-08-24T01:12:21.332 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.333 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.333 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-08-24T01:12:21.333 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:45.955464+0000", 2020-08-24T01:12:21.333 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:58.389375+0000" 2020-08-24T01:12:21.333 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.334 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.334 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-08-24T01:12:21.334 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:58.389375+0000", 2020-08-24T01:12:21.334 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:58.390222+0000" 2020-08-24T01:12:21.334 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.334 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.335 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Backfilling", 2020-08-24T01:12:21.335 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:58.390223+0000", 2020-08-24T01:12:21.335 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:26.789264+0000" 2020-08-24T01:12:21.335 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.335 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.336 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.336 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:26.789265+0000", 2020-08-24T01:12:21.336 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:26.789706+0000" 2020-08-24T01:12:21.336 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.336 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.336 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.337 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:26.789707+0000", 2020-08-24T01:12:21.337 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:27.239063+0000" 2020-08-24T01:12:21.337 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.337 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.337 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.338 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.692936+0000", 2020-08-24T01:12:21.338 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:27.239120+0000" 2020-08-24T01:12:21.338 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.338 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.338 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.339 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.431120+0000", 2020-08-24T01:12:21.339 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:27.239155+0000" 2020-08-24T01:12:21.339 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.339 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.339 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.339 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:15.431063+0000", 2020-08-24T01:12:21.340 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:27.239185+0000" 2020-08-24T01:12:21.340 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.340 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.340 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.340 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.341 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "114", 2020-08-24T01:12:21.341 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.341 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.341 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.341 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:27.239186+0000", 2020-08-24T01:12:21.341 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:27.239745+0000" 2020-08-24T01:12:21.342 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.342 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.342 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.342 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.342 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.343 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.343 INFO:tasks.workunit.client.0.smithi060.stdout: "pg": "3.0", 2020-08-24T01:12:21.343 INFO:tasks.workunit.client.0.smithi060.stdout: "currently": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.343 INFO:tasks.workunit.client.0.smithi060.stdout: "history": [ 2020-08-24T01:12:21.343 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.344 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "39", 2020-08-24T01:12:21.344 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.344 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.344 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.344 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.315238+0000", 2020-08-24T01:12:21.344 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:14.315247+0000" 2020-08-24T01:12:21.345 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.345 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.345 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.345 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.315253+0000", 2020-08-24T01:12:21.345 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:14.468052+0000" 2020-08-24T01:12:21.346 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.346 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.346 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.346 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.468054+0000", 2020-08-24T01:12:21.347 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:14.468229+0000" 2020-08-24T01:12:21.347 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.347 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.347 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.347 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.468230+0000", 2020-08-24T01:12:21.348 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:14.468265+0000" 2020-08-24T01:12:21.348 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.348 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.348 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.348 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.468266+0000", 2020-08-24T01:12:21.348 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:14.742171+0000" 2020-08-24T01:12:21.349 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.349 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.349 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.349 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.315250+0000", 2020-08-24T01:12:21.349 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:14.742180+0000" 2020-08-24T01:12:21.350 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.350 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.350 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.350 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.742334+0000", 2020-08-24T01:12:21.350 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.041188+0000" 2020-08-24T01:12:21.351 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.351 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.351 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.351 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.041190+0000", 2020-08-24T01:12:21.351 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.041472+0000" 2020-08-24T01:12:21.351 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.352 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.352 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.352 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.041474+0000", 2020-08-24T01:12:21.352 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.856804+0000" 2020-08-24T01:12:21.352 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.353 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.353 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.353 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.742180+0000", 2020-08-24T01:12:21.353 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.856862+0000" 2020-08-24T01:12:21.353 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.354 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.354 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.354 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.315248+0000", 2020-08-24T01:12:21.354 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.856897+0000" 2020-08-24T01:12:21.355 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.355 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.356 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.356 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:14.315235+0000", 2020-08-24T01:12:21.356 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.856907+0000" 2020-08-24T01:12:21.356 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.357 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.357 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.357 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.357 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "39", 2020-08-24T01:12:21.357 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.358 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.358 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.358 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.856908+0000", 2020-08-24T01:12:21.358 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.857434+0000" 2020-08-24T01:12:21.359 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.359 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.359 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.359 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.359 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "89", 2020-08-24T01:12:21.359 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.360 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.360 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.360 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.857445+0000", 2020-08-24T01:12:21.360 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.857476+0000" 2020-08-24T01:12:21.361 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.361 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.361 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.361 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.857500+0000", 2020-08-24T01:12:21.362 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.992232+0000" 2020-08-24T01:12:21.362 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.364 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.365 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.365 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.992233+0000", 2020-08-24T01:12:21.365 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.992424+0000" 2020-08-24T01:12:21.366 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.366 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.366 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.367 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.992427+0000", 2020-08-24T01:12:21.367 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:15.992469+0000" 2020-08-24T01:12:21.367 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.367 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.368 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.368 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.992473+0000", 2020-08-24T01:12:21.368 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:16.385035+0000" 2020-08-24T01:12:21.369 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.369 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.370 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.371 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.857491+0000", 2020-08-24T01:12:21.371 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:16.385088+0000" 2020-08-24T01:12:21.372 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.372 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.372 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.372 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:16.385489+0000", 2020-08-24T01:12:21.372 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:16.962818+0000" 2020-08-24T01:12:21.373 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.373 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.373 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.373 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:16.962819+0000", 2020-08-24T01:12:21.374 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:16.962959+0000" 2020-08-24T01:12:21.374 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.374 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.374 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.374 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:16.962959+0000", 2020-08-24T01:12:21.375 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.285937+0000" 2020-08-24T01:12:21.375 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.375 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.375 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.375 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:16.385089+0000", 2020-08-24T01:12:21.376 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.286009+0000" 2020-08-24T01:12:21.376 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.376 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.376 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.377 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.857477+0000", 2020-08-24T01:12:21.377 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.286079+0000" 2020-08-24T01:12:21.377 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.379 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.380 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.380 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:15.857435+0000", 2020-08-24T01:12:21.380 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.286103+0000" 2020-08-24T01:12:21.380 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.380 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.381 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.381 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.381 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "91", 2020-08-24T01:12:21.381 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.381 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.382 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.382 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.286104+0000", 2020-08-24T01:12:21.382 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.287324+0000" 2020-08-24T01:12:21.382 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.382 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.383 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.383 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.383 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "106", 2020-08-24T01:12:21.383 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.383 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.384 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.384 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.287349+0000", 2020-08-24T01:12:21.384 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.287423+0000" 2020-08-24T01:12:21.384 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.384 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.384 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.385 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.287472+0000", 2020-08-24T01:12:21.385 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.401846+0000" 2020-08-24T01:12:21.385 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.385 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.385 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.386 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.401848+0000", 2020-08-24T01:12:21.386 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.402103+0000" 2020-08-24T01:12:21.386 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.386 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.386 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.387 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.402105+0000", 2020-08-24T01:12:21.387 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.402161+0000" 2020-08-24T01:12:21.387 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.387 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.387 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.387 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.402162+0000", 2020-08-24T01:12:21.388 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.718836+0000" 2020-08-24T01:12:21.388 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.388 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.388 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.388 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.287447+0000", 2020-08-24T01:12:21.389 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.718858+0000" 2020-08-24T01:12:21.389 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.389 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.389 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.389 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.719038+0000", 2020-08-24T01:12:21.390 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.848150+0000" 2020-08-24T01:12:21.390 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.390 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.390 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.390 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.848151+0000", 2020-08-24T01:12:21.391 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:51.848190+0000" 2020-08-24T01:12:21.391 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.391 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.391 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.391 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.848191+0000", 2020-08-24T01:12:21.392 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.136887+0000" 2020-08-24T01:12:21.392 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.392 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.392 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.392 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.718858+0000", 2020-08-24T01:12:21.392 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.136931+0000" 2020-08-24T01:12:21.393 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.393 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.393 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.393 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.287424+0000", 2020-08-24T01:12:21.393 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.137007+0000" 2020-08-24T01:12:21.394 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.394 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.394 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.394 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:51.287325+0000", 2020-08-24T01:12:21.394 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.137017+0000" 2020-08-24T01:12:21.395 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.395 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.395 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.395 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.395 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "106", 2020-08-24T01:12:21.395 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.396 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.396 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.396 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.137018+0000", 2020-08-24T01:12:21.396 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.137603+0000" 2020-08-24T01:12:21.396 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.397 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.397 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.397 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.397 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "108", 2020-08-24T01:12:21.397 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.398 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.398 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.398 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.137614+0000", 2020-08-24T01:12:21.398 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.137644+0000" 2020-08-24T01:12:21.398 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.398 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.399 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.399 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.137663+0000", 2020-08-24T01:12:21.399 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.204662+0000" 2020-08-24T01:12:21.399 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.399 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.400 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.400 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.204663+0000", 2020-08-24T01:12:21.400 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.204795+0000" 2020-08-24T01:12:21.400 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.400 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.400 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.401 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.137654+0000", 2020-08-24T01:12:21.401 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.204818+0000" 2020-08-24T01:12:21.401 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.401 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.401 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/WaitActingChange", 2020-08-24T01:12:21.402 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.204818+0000", 2020-08-24T01:12:21.402 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.406090+0000" 2020-08-24T01:12:21.409 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.409 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.410 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.410 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.137645+0000", 2020-08-24T01:12:21.410 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.406132+0000" 2020-08-24T01:12:21.410 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.410 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.410 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.411 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.137604+0000", 2020-08-24T01:12:21.411 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.406146+0000" 2020-08-24T01:12:21.411 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.411 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.411 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.412 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.412 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "108", 2020-08-24T01:12:21.412 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.412 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.412 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.413 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.406147+0000", 2020-08-24T01:12:21.413 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.406767+0000" 2020-08-24T01:12:21.413 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.413 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.413 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.413 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.414 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "116", 2020-08-24T01:12:21.414 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.414 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.414 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.414 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.406782+0000", 2020-08-24T01:12:21.415 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.406825+0000" 2020-08-24T01:12:21.415 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.415 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.415 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.415 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.406852+0000", 2020-08-24T01:12:21.416 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.506304+0000" 2020-08-24T01:12:21.416 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.416 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.416 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.416 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.506305+0000", 2020-08-24T01:12:21.416 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.506561+0000" 2020-08-24T01:12:21.417 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.417 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.417 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.417 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.506562+0000", 2020-08-24T01:12:21.417 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.506611+0000" 2020-08-24T01:12:21.418 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.418 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.418 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.418 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.506611+0000", 2020-08-24T01:12:21.418 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.708481+0000" 2020-08-24T01:12:21.419 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.419 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.419 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.419 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.406839+0000", 2020-08-24T01:12:21.419 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.708514+0000" 2020-08-24T01:12:21.419 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.420 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.420 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.420 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.709177+0000", 2020-08-24T01:12:21.420 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:33.868630+0000" 2020-08-24T01:12:21.420 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.421 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.421 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-08-24T01:12:21.421 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.868633+0000", 2020-08-24T01:12:21.421 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:45.955201+0000" 2020-08-24T01:12:21.421 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.422 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.422 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-08-24T01:12:21.422 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:45.955202+0000", 2020-08-24T01:12:21.422 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:45.956073+0000" 2020-08-24T01:12:21.422 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.422 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.423 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Backfilling", 2020-08-24T01:12:21.423 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:45.956074+0000", 2020-08-24T01:12:21.423 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:58.389041+0000" 2020-08-24T01:12:21.423 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.423 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.424 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-08-24T01:12:21.424 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:58.389043+0000", 2020-08-24T01:12:21.424 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:58.389635+0000" 2020-08-24T01:12:21.424 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.424 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.425 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-08-24T01:12:21.425 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:58.389636+0000", 2020-08-24T01:12:21.425 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:26.790154+0000" 2020-08-24T01:12:21.425 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.426 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.426 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-08-24T01:12:21.426 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:26.790155+0000", 2020-08-24T01:12:21.426 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:26.790947+0000" 2020-08-24T01:12:21.426 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.427 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.427 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Backfilling", 2020-08-24T01:12:21.427 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:26.790948+0000", 2020-08-24T01:12:21.427 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.183355+0000" 2020-08-24T01:12:21.427 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.427 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.428 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.428 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:51.183356+0000", 2020-08-24T01:12:21.428 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.183693+0000" 2020-08-24T01:12:21.428 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.428 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.429 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.429 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:51.183694+0000", 2020-08-24T01:12:21.429 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.466088+0000" 2020-08-24T01:12:21.429 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.429 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.430 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.430 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.708515+0000", 2020-08-24T01:12:21.430 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.466174+0000" 2020-08-24T01:12:21.430 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.430 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.430 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.431 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.406826+0000", 2020-08-24T01:12:21.431 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.466220+0000" 2020-08-24T01:12:21.431 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.431 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.431 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.432 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:33.406768+0000", 2020-08-24T01:12:21.432 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.466237+0000" 2020-08-24T01:12:21.432 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.432 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.432 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.432 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.433 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "116", 2020-08-24T01:12:21.433 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.433 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.433 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.433 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:51.466238+0000", 2020-08-24T01:12:21.434 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.466915+0000" 2020-08-24T01:12:21.434 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.434 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.434 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.434 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.435 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.435 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.435 INFO:tasks.workunit.client.0.smithi060.stdout: "pg": "1.0", 2020-08-24T01:12:21.435 INFO:tasks.workunit.client.0.smithi060.stdout: "currently": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.435 INFO:tasks.workunit.client.0.smithi060.stdout: "history": [ 2020-08-24T01:12:21.435 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.436 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "24", 2020-08-24T01:12:21.436 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.436 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.436 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Initial", 2020-08-24T01:12:21.436 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.002089+0000", 2020-08-24T01:12:21.437 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.002358+0000" 2020-08-24T01:12:21.437 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.437 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.437 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.437 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.438 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "24", 2020-08-24T01:12:21.438 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.438 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.438 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.438 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.002359+0000", 2020-08-24T01:12:21.438 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.002502+0000" 2020-08-24T01:12:21.439 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.439 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.439 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.439 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.439 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "27", 2020-08-24T01:12:21.440 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.440 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.440 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.440 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.002515+0000", 2020-08-24T01:12:21.440 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.002551+0000" 2020-08-24T01:12:21.441 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.441 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.441 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.441 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.002575+0000", 2020-08-24T01:12:21.441 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.119348+0000" 2020-08-24T01:12:21.441 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.442 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.442 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.442 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.119350+0000", 2020-08-24T01:12:21.442 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.119659+0000" 2020-08-24T01:12:21.442 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.443 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.443 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.443 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.119660+0000", 2020-08-24T01:12:21.443 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.119721+0000" 2020-08-24T01:12:21.443 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.444 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.444 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.444 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.119723+0000", 2020-08-24T01:12:21.444 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.269075+0000" 2020-08-24T01:12:21.444 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.444 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.445 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.445 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.002563+0000", 2020-08-24T01:12:21.445 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:09.269095+0000" 2020-08-24T01:12:21.445 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.445 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.446 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.446 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.269649+0000", 2020-08-24T01:12:21.446 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.062566+0000" 2020-08-24T01:12:21.446 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.446 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.447 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.447 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.062567+0000", 2020-08-24T01:12:21.447 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.062705+0000" 2020-08-24T01:12:21.447 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.447 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.448 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.448 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.062706+0000", 2020-08-24T01:12:21.448 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.886318+0000" 2020-08-24T01:12:21.448 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.448 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.449 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.449 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.269096+0000", 2020-08-24T01:12:21.449 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.886368+0000" 2020-08-24T01:12:21.449 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.449 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.450 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.450 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.002552+0000", 2020-08-24T01:12:21.450 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.886409+0000" 2020-08-24T01:12:21.450 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.450 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.450 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.451 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:09.002503+0000", 2020-08-24T01:12:21.451 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.886422+0000" 2020-08-24T01:12:21.451 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.451 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.451 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.452 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.452 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "27", 2020-08-24T01:12:21.452 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.452 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.452 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.453 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.886423+0000", 2020-08-24T01:12:21.453 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.887052+0000" 2020-08-24T01:12:21.453 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.453 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.453 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.453 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.454 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "103", 2020-08-24T01:12:21.454 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.454 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.454 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.454 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.887066+0000", 2020-08-24T01:12:21.455 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.887105+0000" 2020-08-24T01:12:21.455 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.455 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.455 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.455 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.887130+0000", 2020-08-24T01:12:21.456 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.979310+0000" 2020-08-24T01:12:21.456 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.456 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.456 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.456 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.979310+0000", 2020-08-24T01:12:21.456 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.979373+0000" 2020-08-24T01:12:21.457 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.457 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.457 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.457 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.979374+0000", 2020-08-24T01:12:21.457 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:10.979385+0000" 2020-08-24T01:12:21.458 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.458 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.458 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.458 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.979385+0000", 2020-08-24T01:12:21.458 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:11.275147+0000" 2020-08-24T01:12:21.458 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.459 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.459 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.459 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.887118+0000", 2020-08-24T01:12:21.459 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:11.275175+0000" 2020-08-24T01:12:21.459 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.460 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.460 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.460 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:11.275861+0000", 2020-08-24T01:12:21.460 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:11.745152+0000" 2020-08-24T01:12:21.460 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.461 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.461 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.461 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:11.745153+0000", 2020-08-24T01:12:21.461 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:08:11.745297+0000" 2020-08-24T01:12:21.461 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.461 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.462 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.462 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:11.745298+0000", 2020-08-24T01:12:21.462 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.252480+0000" 2020-08-24T01:12:21.462 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.462 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.463 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.463 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:11.275176+0000", 2020-08-24T01:12:21.463 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.252543+0000" 2020-08-24T01:12:21.463 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.463 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.464 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.464 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.887106+0000", 2020-08-24T01:12:21.464 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.252647+0000" 2020-08-24T01:12:21.464 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.464 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.464 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.465 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:08:10.887053+0000", 2020-08-24T01:12:21.465 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.252658+0000" 2020-08-24T01:12:21.465 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.465 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.465 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.466 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.466 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "103", 2020-08-24T01:12:21.466 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.466 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.466 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.466 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.252659+0000", 2020-08-24T01:12:21.467 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.253420+0000" 2020-08-24T01:12:21.467 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.467 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.467 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.467 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.468 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "104", 2020-08-24T01:12:21.468 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.468 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.468 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.468 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.253436+0000", 2020-08-24T01:12:21.468 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.253477+0000" 2020-08-24T01:12:21.469 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.469 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.469 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.469 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.253502+0000", 2020-08-24T01:12:21.469 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.255151+0000" 2020-08-24T01:12:21.470 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.470 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.470 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.470 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.255153+0000", 2020-08-24T01:12:21.470 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.255335+0000" 2020-08-24T01:12:21.471 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.471 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.471 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.471 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.253488+0000", 2020-08-24T01:12:21.471 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.255361+0000" 2020-08-24T01:12:21.471 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.472 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.472 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/WaitActingChange", 2020-08-24T01:12:21.472 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.255362+0000", 2020-08-24T01:12:21.472 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.467073+0000" 2020-08-24T01:12:21.472 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.473 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.473 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.473 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.253478+0000", 2020-08-24T01:12:21.473 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.467156+0000" 2020-08-24T01:12:21.473 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.473 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.474 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.474 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.253421+0000", 2020-08-24T01:12:21.474 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.467184+0000" 2020-08-24T01:12:21.474 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.474 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.475 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.475 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.475 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "104", 2020-08-24T01:12:21.475 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.475 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.476 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.476 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.467185+0000", 2020-08-24T01:12:21.476 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.468393+0000" 2020-08-24T01:12:21.476 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.476 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.476 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.477 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.477 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "118", 2020-08-24T01:12:21.477 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.477 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.477 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Start", 2020-08-24T01:12:21.478 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.468421+0000", 2020-08-24T01:12:21.478 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.468508+0000" 2020-08-24T01:12:21.478 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.478 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.478 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-08-24T01:12:21.479 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.468561+0000", 2020-08-24T01:12:21.479 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.593312+0000" 2020-08-24T01:12:21.479 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.479 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.479 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetLog", 2020-08-24T01:12:21.480 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.593313+0000", 2020-08-24T01:12:21.480 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.593559+0000" 2020-08-24T01:12:21.480 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.480 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.480 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-08-24T01:12:21.481 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.593560+0000", 2020-08-24T01:12:21.481 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.593628+0000" 2020-08-24T01:12:21.481 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.481 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.481 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-08-24T01:12:21.482 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.593629+0000", 2020-08-24T01:12:21.482 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.807537+0000" 2020-08-24T01:12:21.482 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.482 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.482 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Peering", 2020-08-24T01:12:21.483 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.468535+0000", 2020-08-24T01:12:21.483 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:26.807572+0000" 2020-08-24T01:12:21.483 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.483 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.483 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Activating", 2020-08-24T01:12:21.484 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.808353+0000", 2020-08-24T01:12:21.484 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:10:27.094879+0000" 2020-08-24T01:12:21.484 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.484 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.484 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-08-24T01:12:21.485 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:27.094881+0000", 2020-08-24T01:12:21.485 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.184044+0000" 2020-08-24T01:12:21.485 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.485 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.485 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-08-24T01:12:21.486 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:51.184045+0000", 2020-08-24T01:12:21.486 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:11:51.184888+0000" 2020-08-24T01:12:21.486 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.486 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.486 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Backfilling", 2020-08-24T01:12:21.486 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:11:51.184889+0000", 2020-08-24T01:12:21.487 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:12:15.837228+0000" 2020-08-24T01:12:21.487 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.487 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.487 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Recovered", 2020-08-24T01:12:21.487 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:12:15.837246+0000", 2020-08-24T01:12:21.488 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:12:15.837445+0000" 2020-08-24T01:12:21.488 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.488 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.488 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active/Clean", 2020-08-24T01:12:21.488 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:12:15.837446+0000", 2020-08-24T01:12:21.489 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:12:16.048675+0000" 2020-08-24T01:12:21.489 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.489 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.489 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary/Active", 2020-08-24T01:12:21.489 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.807573+0000", 2020-08-24T01:12:21.489 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:12:16.048717+0000" 2020-08-24T01:12:21.490 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.490 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.490 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started/Primary", 2020-08-24T01:12:21.490 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.468509+0000", 2020-08-24T01:12:21.490 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:12:16.048752+0000" 2020-08-24T01:12:21.491 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.491 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.491 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Started", 2020-08-24T01:12:21.491 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:10:26.468394+0000", 2020-08-24T01:12:21.491 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:12:16.048783+0000" 2020-08-24T01:12:21.491 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.492 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.492 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:12:21.492 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.492 INFO:tasks.workunit.client.0.smithi060.stdout: "epoch": "118", 2020-08-24T01:12:21.492 INFO:tasks.workunit.client.0.smithi060.stdout: "states": [ 2020-08-24T01:12:21.493 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:12:21.493 INFO:tasks.workunit.client.0.smithi060.stdout: "state": "Reset", 2020-08-24T01:12:21.493 INFO:tasks.workunit.client.0.smithi060.stdout: "enter": "2020-08-24T01:12:16.048784+0000", 2020-08-24T01:12:21.493 INFO:tasks.workunit.client.0.smithi060.stdout: "exit": "2020-08-24T01:12:16.049399+0000" 2020-08-24T01:12:21.493 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.493 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.494 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.494 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.494 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:12:21.494 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:12:21.494 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:12:21.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-08-24T01:12:21.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-08-24T01:12:21.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-08-24T01:12:21.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-08-24T01:12:21.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-08-24T01:12:21.497 INFO:tasks.workunit.client.0.smithi060.stdout:TEST PASSED 2020-08-24T01:12:21.820 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test1' does not exist 2020-08-24T01:12:21.829 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-08-24T01:12:21.829 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-08-24T01:12:21.830 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-08-24T01:12:22.347 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test3' does not exist 2020-08-24T01:12:22.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-08-24T01:12:22.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-08-24T01:12:22.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-08-24T01:12:22.891 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test7' does not exist 2020-08-24T01:12:22.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-08-24T01:12:22.901 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:12:22.901 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:12:22.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:12:22.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:12:22.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:12:23.018 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:12:23.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-08-24T01:12:23.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-08-24T01:12:23.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-08-24T01:12:23.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:12:23.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-08-24T01:12:23.021 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:12:23.021 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:12:23.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:12:23.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:12:23.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:12:23.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:12:23.034 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:12:23.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:12:23.044 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:12:23.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:12:23.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:12:23.045 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:12:23.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:12:23.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:12:23.045 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:12:23.046 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:12:23.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:12:23.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:12:23.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-08-24T01:12:23.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:12:23.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:23.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:12:23.075 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50053 2020-08-24T01:12:23.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:12:23.077 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:12:23.077 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: code=0 2020-08-24T01:12:23.077 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2154: main: teardown td/osd-backfill-prio 0 2020-08-24T01:12:23.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-08-24T01:12:23.078 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-08-24T01:12:23.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-08-24T01:12:23.079 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:12:23.079 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:12:23.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:12:23.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:12:23.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:12:23.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:12:23.081 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:12:23.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:12:23.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:12:23.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:12:23.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:12:23.087 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:12:23.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:12:23.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:12:23.088 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:12:23.088 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:12:23.089 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:12:23.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-08-24T01:12:23.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-08-24T01:12:23.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:12:23.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:23.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.50053 2020-08-24T01:12:23.095 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.50053 2020-08-24T01:12:23.097 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:12:23.098 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:12:23.098 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2155: main: return 0 2020-08-24T01:12:23.099 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-08-24T01:12:23.100 INFO:teuthology.orchestra.run.smithi060:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-08-24T01:12:23.117 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-08-24T01:12:23.118 INFO:teuthology.orchestra.run.smithi060:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=0a941e55043ccee6d35a8717bec55ff7581e67d8 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-08-24T01:12:23.178 INFO:tasks.workunit.client.0.smithi060.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-08-24T01:12:23.192 INFO:tasks.workunit.client.0.smithi060.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-08-24T01:12:23.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T01:12:23.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2143: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/sbin 2020-08-24T01:12:23.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: export PYTHONWARNINGS=ignore 2020-08-24T01:12:23.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: PYTHONWARNINGS=ignore 2020-08-24T01:12:23.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: export CEPH_CONF=/dev/null 2020-08-24T01:12:23.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: CEPH_CONF=/dev/null 2020-08-24T01:12:23.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: unset CEPH_ARGS 2020-08-24T01:12:23.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: main: local code 2020-08-24T01:12:23.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2149: main: run td/osd-backfill-recovery-log 2020-08-24T01:12:23.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-08-24T01:12:23.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-08-24T01:12:23.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-08-24T01:12:23.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-08-24T01:12:23.195 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-08-24T01:12:23.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none ' 2020-08-24T01:12:23.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:12:23.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-08-24T01:12:23.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-08-24T01:12:23.198 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-08-24T01:12:23.198 INFO:tasks.workunit.client.0.smithi060.stderr:TEST_backfill_log_2 2020-08-24T01:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr:TEST_recovery_1 2020-08-24T01:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr:TEST_recovery_2' 2020-08-24T01:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-08-24T01:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-08-24T01:12:23.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-08-24T01:12:23.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:12:23.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T01:12:23.201 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:12:23.201 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:12:23.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:12:23.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:12:23.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:12:23.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:12:23.203 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:12:23.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:12:23.204 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:12:23.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:12:23.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:12:23.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:12:23.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:12:23.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:12:23.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:12:23.207 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:12:23.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:12:23.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:12:23.210 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T01:12:23.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:12:23.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:23.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:23.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T01:12:23.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:12:23.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:12:23.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-08-24T01:12:23.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:12:23.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:23.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:23.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71396 2020-08-24T01:12:23.221 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:12:23.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-08-24T01:12:23.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-08-24T01:12:23.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:12:23.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-08-24T01:12:23.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-08-24T01:12:23.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:107: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-08-24T01:12:23.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:23.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-08-24T01:12:23.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-08-24T01:12:23.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-08-24T01:12:23.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-08-24T01:12:23.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-08-24T01:12:23.224 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-08-24T01:12:23.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:12:23.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:12:23.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:12:23.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-08-24T01:12:23.226 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:12:23.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:23.526 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:23.526 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:23.527 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:23.528 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:23.528 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:23.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:12:23.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:12:23.739 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:12:23.739 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:12:23.739 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:12:23.740 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:12:23.741 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:12:23.741 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:12:23.741 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:12:23.741 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:12:23.742 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:12:23.742 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:23.742 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:23.742 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:12:23.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:12:23.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get fsid 2020-08-24T01:12:23.838 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:12:23.838 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:12:23.838 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:12:23.838 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:12:23.840 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:12:23.840 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:12:23.840 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:12:23.841 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:12:23.841 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:12:23.841 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:23.841 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:23.842 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:12:23.842 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:12:23.843 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get mon_host 2020-08-24T01:12:23.931 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-08-24T01:12:23.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:23.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:12:23.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:12:23.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:12:23.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-08-24T01:12:23.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:12:24.477 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:12:24.477 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:24.478 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:24.478 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:24.479 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:24.479 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:24.479 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:24.480 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:12:24.481 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:12:24.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-08-24T01:12:24.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:24.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:24.507 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-08-24T01:12:24.508 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-08-24T01:12:24.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:12:24.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-08-24T01:12:24.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:24.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:12:24.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:12:24.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:12:24.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-08-24T01:12:24.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:12:24.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:12:24.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:12:24.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:12:24.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-08-24T01:12:24.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-08-24T01:12:24.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:12:24.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:24.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:12:24.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:12:24.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:24.515 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:24.517 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:24.518 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:24.518 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:24.518 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:24.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:24.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:12:24.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:12:24.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:12:24.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:12:24.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:12:24.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:12:24.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:12:24.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:24.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:12:24.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:12:24.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-08-24T01:12:24.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:12:24.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=56948807-7939-419d-b258-b935d1a60169 2020-08-24T01:12:24.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 56948807-7939-419d-b258-b935d1a60169' 2020-08-24T01:12:24.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:12:24.522 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 56948807-7939-419d-b258-b935d1a60169 2020-08-24T01:12:24.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD4E0Nf1Ae9HxAARJcrSBCM9enLV4fLRcDFVA== 2020-08-24T01:12:24.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD4E0Nf1Ae9HxAARJcrSBCM9enLV4fLRcDFVA=="}' 2020-08-24T01:12:24.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 56948807-7939-419d-b258-b935d1a60169 -i td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:12:24.868 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:12:24.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:12:24.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD4E0Nf1Ae9HxAARJcrSBCM9enLV4fLRcDFVA== --osd-uuid 56948807-7939-419d-b258-b935d1a60169 2020-08-24T01:12:24.912 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:24.903+0000 7fedb77ded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:24.915 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:24.907+0000 7fedb77ded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:24.916 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:24.911+0000 7fedb77ded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:24.917 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:24.911+0000 7fedb77ded80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:12:24.917 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:24.911+0000 7fedb77ded80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:12:24.917 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:24.911+0000 7fedb77ded80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:12:24.954 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:24.947+0000 7fedb77ded80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-08-24T01:12:25.348 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:25.339+0000 7fedb77ded80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:12:28.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-08-24T01:12:28.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:12:28.087 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:12:28.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:12:28.088 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:12:28.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:12:28.553 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:12:28.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:28.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:12:28.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:12:28.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:12:28.582 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:28.575+0000 7ff954853d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:28.584 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:28.579+0000 7ff954853d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:28.585 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:28.579+0000 7ff954853d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:28.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:12:28.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:12:28.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:12:28.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:12:28.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:12:28.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:28.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:12:28.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:28.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:12:28.923 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:12:29.129 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:29.123+0000 7ff954853d80 -1 Falling back to public interface 2020-08-24T01:12:29.275 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:12:29.908 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:29.903+0000 7ff954853d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:12:30.277 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:12:30.277 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:30.277 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:12:30.277 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:12:30.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:30.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:12:30.639 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/71821,v1:127.0.0.1:6803/71821] [v2:127.0.0.1:6804/71821,v1:127.0.0.1:6805/71821] exists,up 56948807-7939-419d-b258-b935d1a60169 2020-08-24T01:12:30.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:12:30.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:12:30.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:12:30.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:12:30.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-08-24T01:12:30.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:30.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:12:30.644 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:12:30.644 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:12:30.644 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-08-24T01:12:30.644 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:12:30.644 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:12:30.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:12:30.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:12:30.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-08-24T01:12:30.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-08-24T01:12:30.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:12:30.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:30.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:12:30.646 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:12:30.646 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:30.646 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:30.647 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:30.647 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:30.647 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:30.647 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:30.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:30.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:12:30.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:12:30.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:12:30.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:12:30.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:12:30.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:12:30.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:12:30.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:30.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:12:30.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:12:30.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-08-24T01:12:30.652 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:12:30.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=688d66f8-3225-4959-83f1-2e85a910ef76 2020-08-24T01:12:30.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 688d66f8-3225-4959-83f1-2e85a910ef76' 2020-08-24T01:12:30.653 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:12:30.653 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 688d66f8-3225-4959-83f1-2e85a910ef76 2020-08-24T01:12:30.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD+E0NfrM6rJxAAsZjCAuSv4Loq0synxsWEgQ== 2020-08-24T01:12:30.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD+E0NfrM6rJxAAsZjCAuSv4Loq0synxsWEgQ=="}' 2020-08-24T01:12:30.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 688d66f8-3225-4959-83f1-2e85a910ef76 -i td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:12:31.227 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:12:31.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:12:31.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD+E0NfrM6rJxAAsZjCAuSv4Loq0synxsWEgQ== --osd-uuid 688d66f8-3225-4959-83f1-2e85a910ef76 2020-08-24T01:12:31.271 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.263+0000 7f4de9879d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:31.272 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.267+0000 7f4de9879d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:31.274 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.267+0000 7f4de9879d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:31.275 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.267+0000 7f4de9879d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:12:31.275 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.267+0000 7f4de9879d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:12:31.275 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.267+0000 7f4de9879d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:12:31.357 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.351+0000 7f4de9879d80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-08-24T01:12:31.806 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:31.803+0000 7f4de9879d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:12:34.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-08-24T01:12:34.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:12:34.429 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:12:34.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:12:34.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:12:34.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:12:34.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:12:34.910 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:12:34.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:12:34.911 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:34.911 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:12:34.940 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:34.935+0000 7fa5d2d5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:34.943 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:34.935+0000 7fa5d2d5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:34.944 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:34.939+0000 7fa5d2d5cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:35.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:12:35.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:12:35.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:12:35.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:12:35.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:12:35.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:35.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:12:35.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:35.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:12:35.255 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:12:35.482 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:35.475+0000 7fa5d2d5cd80 -1 Falling back to public interface 2020-08-24T01:12:35.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:12:36.322 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:36.311+0000 7fa5d2d5cd80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:12:36.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:12:36.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:36.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:12:36.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:36.595 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:12:36.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:12:36.998 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/72176,v1:127.0.0.1:6811/72176] [v2:127.0.0.1:6812/72176,v1:127.0.0.1:6813/72176] exists,up 688d66f8-3225-4959-83f1-2e85a910ef76 2020-08-24T01:12:36.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:12:36.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:12:36.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:12:36.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:12:36.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-08-24T01:12:37.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:37.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:12:37.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T01:12:37.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:12:37.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-08-24T01:12:37.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:12:37.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:12:37.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:12:37.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:12:37.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-08-24T01:12:37.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-08-24T01:12:37.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:12:37.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:37.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:12:37.007 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:12:37.007 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:37.008 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:37.008 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:37.008 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:37.008 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:37.008 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:37.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:37.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:12:37.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:12:37.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:12:37.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:12:37.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:12:37.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:12:37.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:12:37.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:37.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:12:37.011 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:12:37.011 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-08-24T01:12:37.011 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:12:37.011 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8a00669b-53cd-4b0a-8988-d6e2f34a34d3 2020-08-24T01:12:37.011 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 8a00669b-53cd-4b0a-8988-d6e2f34a34d3' 2020-08-24T01:12:37.012 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:12:37.014 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 8a00669b-53cd-4b0a-8988-d6e2f34a34d3 2020-08-24T01:12:37.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAFFENf1/qXARAAWTbsKDgTJ1uEg+qzvHPIrg== 2020-08-24T01:12:37.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAFFENf1/qXARAAWTbsKDgTJ1uEg+qzvHPIrg=="}' 2020-08-24T01:12:37.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8a00669b-53cd-4b0a-8988-d6e2f34a34d3 -i td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:12:37.627 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:12:37.638 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:12:37.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAFFENf1/qXARAAWTbsKDgTJ1uEg+qzvHPIrg== --osd-uuid 8a00669b-53cd-4b0a-8988-d6e2f34a34d3 2020-08-24T01:12:37.672 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:37.667+0000 7f7d7e983d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:37.674 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:37.667+0000 7f7d7e983d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:37.676 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:37.671+0000 7f7d7e983d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:37.676 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:37.671+0000 7f7d7e983d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:12:37.677 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:37.671+0000 7f7d7e983d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:12:37.677 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:37.671+0000 7f7d7e983d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:12:37.728 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:37.727+0000 7f7d7e983d80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-08-24T01:12:38.072 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:38.067+0000 7f7d7e983d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:12:40.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-08-24T01:12:40.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:12:40.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T01:12:40.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:12:40.894 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T01:12:41.359 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T01:12:41.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:41.360 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:12:41.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:12:41.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:12:41.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:12:41.382 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:41.375+0000 7fd758262d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:41.384 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:41.379+0000 7fd758262d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:41.385 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:41.379+0000 7fd758262d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:41.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T01:12:41.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:12:41.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:12:41.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:12:41.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:12:41.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:41.707 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:12:41.707 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:41.707 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:12:41.708 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:12:41.922 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:41.915+0000 7fd758262d80 -1 Falling back to public interface 2020-08-24T01:12:42.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:12:42.733 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:42.727+0000 7fd758262d80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T01:12:43.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:12:43.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:43.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:12:43.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:43.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:12:43.052 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:12:43.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:12:44.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:12:44.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:44.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:12:44.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:44.416 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:12:44.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:12:44.794 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/72531,v1:127.0.0.1:6819/72531] [v2:127.0.0.1:6820/72531,v1:127.0.0.1:6821/72531] exists,up 8a00669b-53cd-4b0a-8988-d6e2f34a34d3 2020-08-24T01:12:44.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:12:44.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:12:44.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:12:44.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:12:44.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-08-24T01:12:44.796 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:44.796 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:12:44.796 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-08-24T01:12:44.796 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:12:44.796 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-08-24T01:12:44.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:12:44.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:12:44.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:12:44.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:12:44.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-08-24T01:12:44.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-08-24T01:12:44.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:12:44.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:44.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:12:44.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:12:44.801 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:44.801 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:44.801 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:44.801 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:44.801 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:44.802 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:44.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:44.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:12:44.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:12:44.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:12:44.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:12:44.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:12:44.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:12:44.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:12:44.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:44.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:12:44.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:12:44.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-08-24T01:12:44.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:12:44.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=438d27a5-b4fb-4269-b7ab-caa53d8b58dc 2020-08-24T01:12:44.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 438d27a5-b4fb-4269-b7ab-caa53d8b58dc' 2020-08-24T01:12:44.806 INFO:tasks.workunit.client.0.smithi060.stdout:add osd3 438d27a5-b4fb-4269-b7ab-caa53d8b58dc 2020-08-24T01:12:44.806 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:12:44.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAMFENf1HY9MRAAnXbOWXhWR1pC7FH6jRtuZA== 2020-08-24T01:12:44.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAMFENf1HY9MRAAnXbOWXhWR1pC7FH6jRtuZA=="}' 2020-08-24T01:12:44.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 438d27a5-b4fb-4269-b7ab-caa53d8b58dc -i td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:12:45.343 INFO:tasks.workunit.client.0.smithi060.stdout:3 2020-08-24T01:12:45.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:12:45.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAMFENf1HY9MRAAnXbOWXhWR1pC7FH6jRtuZA== --osd-uuid 438d27a5-b4fb-4269-b7ab-caa53d8b58dc 2020-08-24T01:12:45.391 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.387+0000 7fdaa3df2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:45.393 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.391+0000 7fdaa3df2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:45.395 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.391+0000 7fdaa3df2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:45.396 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.395+0000 7fdaa3df2d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:12:45.396 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.395+0000 7fdaa3df2d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:12:45.396 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.395+0000 7fdaa3df2d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:12:45.437 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.435+0000 7fdaa3df2d80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-08-24T01:12:45.842 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:45.839+0000 7fdaa3df2d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:12:48.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-08-24T01:12:48.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:12:48.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-08-24T01:12:48.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:12:48.576 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd3 key to auth repository 2020-08-24T01:12:49.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-08-24T01:12:49.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:49.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:12:49.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:12:49.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:12:49.044 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.3 2020-08-24T01:12:49.073 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:49.071+0000 7f05251e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:49.076 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:49.075+0000 7f05251e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:49.077 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:49.075+0000 7f05251e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:49.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-08-24T01:12:49.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:12:49.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-08-24T01:12:49.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:12:49.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:12:49.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:49.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:12:49.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:49.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:12:49.408 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:12:49.610 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:49.607+0000 7f05251e9d80 -1 Falling back to public interface 2020-08-24T01:12:49.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:12:50.386 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:50.383+0000 7f05251e9d80 -1 osd.3 0 log_to_monitors {default=true} 2020-08-24T01:12:50.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:12:50.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:50.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:12:50.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:50.772 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:12:50.772 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:12:51.185 INFO:tasks.workunit.client.0.smithi060.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/72914,v1:127.0.0.1:6827/72914] [v2:127.0.0.1:6828/72914,v1:127.0.0.1:6829/72914] exists,up 438d27a5-b4fb-4269-b7ab-caa53d8b58dc 2020-08-24T01:12:51.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:12:51.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:12:51.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:12:51.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:12:51.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-08-24T01:12:51.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:51.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:12:51.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-08-24T01:12:51.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:12:51.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-08-24T01:12:51.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:12:51.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:12:51.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:12:51.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:12:51.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-08-24T01:12:51.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-08-24T01:12:51.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:12:51.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:51.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:12:51.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:12:51.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:51.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:51.191 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:51.191 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:51.191 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:51.191 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:51.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:51.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:12:51.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:12:51.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:12:51.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:12:51.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:12:51.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:12:51.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:12:51.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:51.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:12:51.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:12:51.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-08-24T01:12:51.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:12:51.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=eadeb041-35a0-452d-a851-8dec12d2b25e 2020-08-24T01:12:51.198 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 eadeb041-35a0-452d-a851-8dec12d2b25e' 2020-08-24T01:12:51.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:12:51.198 INFO:tasks.workunit.client.0.smithi060.stdout:add osd4 eadeb041-35a0-452d-a851-8dec12d2b25e 2020-08-24T01:12:51.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQATFENfSHk6DBAA+/iiqtNxyyFG/M9z+6vw4g== 2020-08-24T01:12:51.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQATFENfSHk6DBAA+/iiqtNxyyFG/M9z+6vw4g=="}' 2020-08-24T01:12:51.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new eadeb041-35a0-452d-a851-8dec12d2b25e -i td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:12:51.798 INFO:tasks.workunit.client.0.smithi060.stdout:4 2020-08-24T01:12:51.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:12:51.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQATFENfSHk6DBAA+/iiqtNxyyFG/M9z+6vw4g== --osd-uuid eadeb041-35a0-452d-a851-8dec12d2b25e 2020-08-24T01:12:51.848 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:51.847+0000 7f75a292ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:51.851 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:51.847+0000 7f75a292ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:51.858 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:51.851+0000 7f75a292ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:51.859 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:51.851+0000 7f75a292ad80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:12:51.859 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:51.851+0000 7f75a292ad80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:12:51.859 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:51.851+0000 7f75a292ad80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:12:51.902 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:51.899+0000 7f75a292ad80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-08-24T01:12:52.330 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:52.323+0000 7f75a292ad80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:12:55.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-08-24T01:12:55.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:12:55.117 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-08-24T01:12:55.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:12:55.118 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd4 key to auth repository 2020-08-24T01:12:55.580 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-08-24T01:12:56.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:56.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:12:56.080 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.4 2020-08-24T01:12:56.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:12:56.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:12:56.082 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:55.607+0000 7f6018a63d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:56.083 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:55.611+0000 7f6018a63d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:56.083 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:55.611+0000 7f6018a63d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:56.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-08-24T01:12:56.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:12:56.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-08-24T01:12:56.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:12:56.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:12:56.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:56.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:12:56.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:56.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:12:56.085 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:12:56.156 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:56.155+0000 7f6018a63d80 -1 Falling back to public interface 2020-08-24T01:12:56.314 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:12:56.947 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:56.935+0000 7f6018a63d80 -1 osd.4 0 log_to_monitors {default=true} 2020-08-24T01:12:57.314 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:12:57.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:57.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:12:57.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:57.316 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:12:57.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:12:57.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:12:58.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:12:58.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:12:58.681 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:12:58.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:12:58.682 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:12:58.683 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:12:59.057 INFO:tasks.workunit.client.0.smithi060.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/73269,v1:127.0.0.1:6835/73269] [v2:127.0.0.1:6836/73269,v1:127.0.0.1:6837/73269] exists,up eadeb041-35a0-452d-a851-8dec12d2b25e 2020-08-24T01:12:59.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:12:59.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:12:59.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:12:59.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:12:59.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-08-24T01:12:59.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:12:59.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:12:59.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-08-24T01:12:59.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:12:59.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-08-24T01:12:59.059 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:12:59.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:12:59.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:12:59.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:12:59.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-08-24T01:12:59.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-08-24T01:12:59.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:12:59.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-08-24T01:12:59.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:12:59.061 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:12:59.061 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:12:59.062 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:12:59.062 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:12:59.062 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:12:59.062 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:12:59.062 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:59.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:12:59.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:12:59.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:12:59.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:12:59.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:12:59.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:12:59.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:12:59.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:12:59.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:12:59.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:12:59.065 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:12:59.066 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-08-24T01:12:59.067 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:12:59.067 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=71898a10-9b65-450f-8d72-e73749494d4d 2020-08-24T01:12:59.067 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 71898a10-9b65-450f-8d72-e73749494d4d' 2020-08-24T01:12:59.067 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:12:59.068 INFO:tasks.workunit.client.0.smithi060.stdout:add osd5 71898a10-9b65-450f-8d72-e73749494d4d 2020-08-24T01:12:59.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAbFENf/K6wBBAAx6RHF4davDJS28HMkmq7KA== 2020-08-24T01:12:59.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAbFENf/K6wBBAAx6RHF4davDJS28HMkmq7KA=="}' 2020-08-24T01:12:59.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 71898a10-9b65-450f-8d72-e73749494d4d -i td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:12:59.570 INFO:tasks.workunit.client.0.smithi060.stdout:5 2020-08-24T01:12:59.581 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:12:59.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAbFENf/K6wBBAAx6RHF4davDJS28HMkmq7KA== --osd-uuid 71898a10-9b65-450f-8d72-e73749494d4d 2020-08-24T01:12:59.617 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:59.615+0000 7fab98d87d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:59.621 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:59.619+0000 7fab98d87d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:59.623 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:59.619+0000 7fab98d87d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:12:59.624 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:59.619+0000 7fab98d87d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:12:59.624 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:59.619+0000 7fab98d87d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:12:59.625 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:59.619+0000 7fab98d87d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:12:59.693 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:12:59.691+0000 7fab98d87d80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-08-24T01:13:00.101 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:13:00.099+0000 7fab98d87d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:13:03.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-08-24T01:13:03.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:13:03.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-08-24T01:13:03.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:13:03.122 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd5 key to auth repository 2020-08-24T01:13:03.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-08-24T01:13:03.595 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.5 2020-08-24T01:13:03.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:13:03.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:13:03.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:13:03.597 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:13:03.626 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:13:03.623+0000 7fbc6bb71d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:13:03.628 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:13:03.627+0000 7fbc6bb71d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:13:03.629 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:13:03.627+0000 7fbc6bb71d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:13:03.956 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-08-24T01:13:03.956 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:13:03.962 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-08-24T01:13:03.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:13:03.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:13:03.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:13:03.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:13:03.963 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:13:03.964 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:13:03.964 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:13:04.162 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:13:04.159+0000 7fbc6bb71d80 -1 Falling back to public interface 2020-08-24T01:13:04.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:13:04.912 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:13:04.911+0000 7fbc6bb71d80 -1 osd.5 0 log_to_monitors {default=true} 2020-08-24T01:13:05.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:13:05.328 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:13:05.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:13:05.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:13:05.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:13:05.330 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:13:05.695 INFO:tasks.workunit.client.0.smithi060.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/73652,v1:127.0.0.1:6843/73652] [v2:127.0.0.1:6844/73652,v1:127.0.0.1:6845/73652] exists,up 71898a10-9b65-450f-8d72-e73749494d4d 2020-08-24T01:13:05.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:13:05.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:13:05.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:13:05.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-08-24T01:13:05.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-08-24T01:13:06.318 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T01:13:06.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:13:07.331 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-08-24T01:13:07.332 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:07.332 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-08-24T01:13:07.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:07.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-08-24T01:13:07.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:07.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-08-24T01:13:07.998 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:07.999 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-08-24T01:13:08.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:08.163 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-08-24T01:13:08.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:08.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-08-24T01:13:08.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:08.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-08-24T01:13:08.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:08.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-08-24T01:13:08.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:08.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-08-24T01:13:08.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:08.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-08-24T01:13:09.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:09.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-08-24T01:13:09.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:09.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-08-24T01:13:09.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:09.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-08-24T01:13:09.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:09.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-08-24T01:13:09.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:09.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-08-24T01:13:09.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:09.800 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-08-24T01:13:09.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:09.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-08-24T01:13:10.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:10.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-08-24T01:13:10.252 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:10.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-08-24T01:13:10.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:10.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-08-24T01:13:10.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:10.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-08-24T01:13:10.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:10.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-08-24T01:13:10.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:10.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-08-24T01:13:10.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:10.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-08-24T01:13:11.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:11.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-08-24T01:13:11.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:11.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-08-24T01:13:11.333 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:11.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-08-24T01:13:11.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:11.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-08-24T01:13:11.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:11.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-08-24T01:13:11.761 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:11.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-08-24T01:13:11.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:11.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-08-24T01:13:12.054 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:12.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-08-24T01:13:12.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:12.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-08-24T01:13:12.365 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:12.365 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-08-24T01:13:12.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:12.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-08-24T01:13:12.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:12.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-08-24T01:13:12.849 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:12.849 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-08-24T01:13:12.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:12.993 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-08-24T01:13:13.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:13.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-08-24T01:13:13.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:13.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-08-24T01:13:13.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:13.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-08-24T01:13:13.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:13.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-08-24T01:13:13.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:13.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-08-24T01:13:13.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:13.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-08-24T01:13:13.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:13.949 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-08-24T01:13:14.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:14.084 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-08-24T01:13:14.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:14.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-08-24T01:13:14.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:14.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-08-24T01:13:14.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:14.528 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-08-24T01:13:14.674 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:14.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-08-24T01:13:14.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:14.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-08-24T01:13:14.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:14.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-08-24T01:13:15.098 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:15.099 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-08-24T01:13:15.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:15.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-08-24T01:13:15.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:15.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-08-24T01:13:15.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:15.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-08-24T01:13:15.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:15.680 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-08-24T01:13:15.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:15.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-08-24T01:13:15.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:15.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-08-24T01:13:16.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:16.080 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-08-24T01:13:16.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:16.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-08-24T01:13:16.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:16.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-08-24T01:13:16.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:16.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-08-24T01:13:16.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:16.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-08-24T01:13:16.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:16.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-08-24T01:13:16.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:16.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-08-24T01:13:17.025 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:17.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-08-24T01:13:17.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:17.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-08-24T01:13:17.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:17.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-08-24T01:13:17.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:17.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-08-24T01:13:17.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:17.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-08-24T01:13:17.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:17.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-08-24T01:13:17.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:17.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-08-24T01:13:18.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:18.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-08-24T01:13:18.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:18.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-08-24T01:13:18.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:18.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-08-24T01:13:18.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:18.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-08-24T01:13:18.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:18.742 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-08-24T01:13:18.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:18.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-08-24T01:13:19.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:19.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-08-24T01:13:19.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:19.187 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-08-24T01:13:19.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:19.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-08-24T01:13:19.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:19.447 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-08-24T01:13:19.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:19.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-08-24T01:13:19.731 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:19.731 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-08-24T01:13:19.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:19.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-08-24T01:13:20.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-08-24T01:13:20.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-08-24T01:13:20.292 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.292 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-08-24T01:13:20.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-08-24T01:13:20.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-08-24T01:13:20.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-08-24T01:13:20.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-08-24T01:13:20.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:20.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-08-24T01:13:21.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:21.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-08-24T01:13:21.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:21.251 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-08-24T01:13:21.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:21.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-08-24T01:13:21.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-08-24T01:13:21.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:21.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-08-24T01:13:21.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:21.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-08-24T01:13:21.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:21.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-08-24T01:13:22.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:22.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-08-24T01:13:22.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:22.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-08-24T01:13:22.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:22.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-08-24T01:13:22.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:22.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-08-24T01:13:22.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:22.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-08-24T01:13:22.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:22.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-08-24T01:13:22.957 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:22.958 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-08-24T01:13:23.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:23.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-08-24T01:13:23.265 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:23.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-08-24T01:13:23.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:23.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-08-24T01:13:23.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:23.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-08-24T01:13:23.670 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:23.670 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-08-24T01:13:23.812 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:23.812 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-08-24T01:13:23.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-08-24T01:13:24.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-08-24T01:13:24.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-08-24T01:13:24.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-08-24T01:13:24.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-08-24T01:13:24.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-08-24T01:13:24.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-08-24T01:13:24.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:24.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-08-24T01:13:25.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:25.103 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-08-24T01:13:25.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:25.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-08-24T01:13:25.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:25.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-08-24T01:13:25.504 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:25.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-08-24T01:13:25.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:25.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-08-24T01:13:25.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:25.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-08-24T01:13:25.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:25.912 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-08-24T01:13:26.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:26.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-08-24T01:13:26.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:26.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-08-24T01:13:26.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:26.342 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-08-24T01:13:26.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:26.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-08-24T01:13:26.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:26.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-08-24T01:13:26.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:26.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-08-24T01:13:26.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:26.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-08-24T01:13:27.088 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:27.088 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-08-24T01:13:27.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:27.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-08-24T01:13:27.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:27.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-08-24T01:13:27.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:27.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-08-24T01:13:27.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:27.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-08-24T01:13:27.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:27.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-08-24T01:13:27.951 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:27.951 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-08-24T01:13:28.094 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:28.094 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-08-24T01:13:28.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:28.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-08-24T01:13:28.571 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:28.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-08-24T01:13:28.704 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:28.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-08-24T01:13:28.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:28.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-08-24T01:13:28.981 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:28.982 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-08-24T01:13:29.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:13:29.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-08-24T01:13:29.266 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-08-24T01:13:29.267 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-08-24T01:13:29.612 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:13:29.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-08-24T01:13:30.120 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-08-24T01:13:30.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-08-24T01:13:30.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-08-24T01:13:31.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-08-24T01:13:31.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:13:31.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:13:31.133 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:13:31.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:13:31.137 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:13:31.137 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:13:31.138 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:13:31.138 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:13:31.138 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:13:31.138 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:13:31.340 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:13:31.340 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:13:31.340 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:13:31.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:13:31.342 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:13:31.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:13:31.776 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:13:31.777 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:13:31.777 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:13:31.778 INFO:tasks.workunit.client.0.smithi060.stderr:4 2020-08-24T01:13:31.778 INFO:tasks.workunit.client.0.smithi060.stderr:5' 2020-08-24T01:13:31.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:13:31.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:13:31.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:13:31.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836495 2020-08-24T01:13:31.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836495 2020-08-24T01:13:31.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495' 2020-08-24T01:13:31.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:13:31.956 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:13:32.104 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672973 2020-08-24T01:13:32.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672973 2020-08-24T01:13:32.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672973' 2020-08-24T01:13:32.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:13:32.106 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:13:32.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509452 2020-08-24T01:13:32.308 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509452 2020-08-24T01:13:32.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509452' 2020-08-24T01:13:32.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:13:32.310 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:13:32.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345931 2020-08-24T01:13:32.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 85899345931 2020-08-24T01:13:32.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509452 3-85899345931' 2020-08-24T01:13:32.475 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:13:32.476 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:13:32.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182409 2020-08-24T01:13:32.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 107374182409 2020-08-24T01:13:32.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509452 3-85899345931 4-107374182409' 2020-08-24T01:13:32.654 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:13:32.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-08-24T01:13:32.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018888 2020-08-24T01:13:32.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 128849018888 2020-08-24T01:13:32.815 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-42949672973 2-64424509452 3-85899345931 4-107374182409 5-128849018888' 2020-08-24T01:13:32.815 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:13:32.818 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836495 2020-08-24T01:13:32.818 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:13:32.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:13:32.827 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2020-08-24T01:13:32.827 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:13:32.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836495 2020-08-24T01:13:32.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-08-24T01:13:32.828 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:13:32.829 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836495 2020-08-24T01:13:33.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836495 -lt 21474836495 2020-08-24T01:13:33.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:13:33.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672973 2020-08-24T01:13:33.180 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:13:33.181 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:13:33.184 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672973 2020-08-24T01:13:33.185 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:13:33.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672973 2020-08-24T01:13:33.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2020-08-24T01:13:33.186 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672973 2020-08-24T01:13:33.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:13:33.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 0 -lt 42949672973 2020-08-24T01:13:33.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:13:34.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:13:34.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:13:34.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672974 -lt 42949672973 2020-08-24T01:13:34.948 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:13:34.949 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509452 2020-08-24T01:13:34.950 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:13:34.953 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:13:34.954 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509452 2020-08-24T01:13:34.954 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:13:34.957 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509452 2020-08-24T01:13:34.957 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509452' 2020-08-24T01:13:34.958 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509452 2020-08-24T01:13:34.958 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:13:35.324 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509452 -lt 64424509452 2020-08-24T01:13:35.324 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:13:35.331 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-85899345931 2020-08-24T01:13:35.334 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:13:35.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:13:35.334 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345931 2020-08-24T01:13:35.335 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:13:35.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=85899345931 2020-08-24T01:13:35.336 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 85899345931' 2020-08-24T01:13:35.336 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:13:35.336 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 85899345931 2020-08-24T01:13:35.697 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 85899345931 -lt 85899345931 2020-08-24T01:13:35.698 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:13:35.698 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:13:35.699 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-107374182409 2020-08-24T01:13:35.699 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:13:35.702 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182409 2020-08-24T01:13:35.702 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:13:35.704 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 107374182409 2020-08-24T01:13:35.704 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=107374182409 2020-08-24T01:13:35.704 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 107374182409' 2020-08-24T01:13:35.708 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:13:36.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 107374182410 -lt 107374182409 2020-08-24T01:13:36.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:13:36.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 5-128849018888 2020-08-24T01:13:36.131 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:13:36.132 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=5 2020-08-24T01:13:36.135 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018888 2020-08-24T01:13:36.135 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:13:36.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=128849018888 2020-08-24T01:13:36.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.5 seq 128849018888' 2020-08-24T01:13:36.137 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.5 seq 128849018888 2020-08-24T01:13:36.139 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 5 2020-08-24T01:13:36.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 128849018888 -lt 128849018888 2020-08-24T01:13:36.510 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:13:36.511 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:36.511 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:36.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:13:36.942 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:36.944 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:36.944 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:36.946 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:36.946 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:36.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:36.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:37.283 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:37.284 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:37.285 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:37.285 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:37.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:37.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T01:13:37.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:13:37.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T01:13:37.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:37.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:37.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:37.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:37.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:37.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:37.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:37.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:37.894 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:37.894 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:38.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:38.247 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:38.247 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:38.247 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:38.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:38.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:38.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:38.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:38.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:38.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:38.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:38.698 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:38.699 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:39.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=12584 2020-08-24T01:13:39.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 12584 '!=' null 2020-08-24T01:13:39.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:39.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:39.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:39.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:39.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:39.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:39.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:39.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:39.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:39.270 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:39.270 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:39.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:39.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:39.624 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:39.624 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:40.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:40.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:40.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:40.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:40.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:40.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:40.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:40.074 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:40.075 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:40.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=9940 2020-08-24T01:13:40.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 9940 '!=' null 2020-08-24T01:13:40.513 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:40.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:40.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:40.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:40.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:40.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:40.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:40.618 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:40.618 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:40.618 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:40.619 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:40.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:40.975 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:40.975 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:40.975 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:41.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:41.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:41.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:41.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:41.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:41.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:41.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:41.414 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:41.414 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:41.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=9940 2020-08-24T01:13:41.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 9940 '!=' null 2020-08-24T01:13:41.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:41.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:41.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:41.962 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:41.962 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:41.965 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:41.965 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:41.965 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:41.965 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:41.966 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:41.966 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:42.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:42.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:42.314 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:42.315 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:42.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:42.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:42.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:42.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:42.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:42.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:42.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:42.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:42.779 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:43.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24213 2020-08-24T01:13:43.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24213 '!=' null 2020-08-24T01:13:43.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:43.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:43.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:43.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:43.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:43.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:43.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:43.337 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:43.338 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:43.338 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:43.338 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:43.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:43.697 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:43.697 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:43.698 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:44.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:44.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:44.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:44.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:44.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:44.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:44.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:44.181 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:44.181 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:44.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20235 2020-08-24T01:13:44.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20235 '!=' null 2020-08-24T01:13:44.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:44.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:44.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:44.721 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:44.721 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:44.721 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:44.722 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:44.722 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:44.722 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:44.722 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:44.722 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:45.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:45.080 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:45.081 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:45.081 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:45.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:45.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:45.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:45.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:45.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:45.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:45.524 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:45.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:45.525 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:46.025 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20235 2020-08-24T01:13:46.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20235 '!=' null 2020-08-24T01:13:46.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:46.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:46.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:46.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:46.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:46.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:46.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:46.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:46.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:46.130 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:46.130 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:46.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:46.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:46.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:46.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:46.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:46.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:46.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:46.959 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:46.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:46.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:46.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:46.960 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:46.960 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:47.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20235 2020-08-24T01:13:47.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20235 '!=' null 2020-08-24T01:13:47.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:47.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:47.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:47.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:47.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:47.553 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:47.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:47.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:47.554 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:47.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:47.555 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:47.912 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:47.914 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:47.915 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:47.915 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:48.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:48.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:48.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:48.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:48.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:48.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:48.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:48.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:48.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:48.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33516 2020-08-24T01:13:48.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33516 '!=' null 2020-08-24T01:13:48.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:48.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:48.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:48.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:48.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:48.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:48.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:48.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:48.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:48.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:48.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:49.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:49.304 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:49.304 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:49.305 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:49.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:49.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:49.822 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:49.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:49.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:49.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:49.823 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:49.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:49.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:50.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26559 2020-08-24T01:13:50.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26559 '!=' null 2020-08-24T01:13:50.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:50.300 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:50.300 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:50.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:50.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:50.390 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:50.390 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:50.390 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:50.391 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:50.391 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:50.391 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:50.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:50.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:50.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:50.736 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:51.177 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:51.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:51.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:51.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:51.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:51.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:51.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:51.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:51.180 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:51.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26559 2020-08-24T01:13:51.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26559 '!=' null 2020-08-24T01:13:51.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:51.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:51.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:51.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:51.726 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:51.734 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:51.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:51.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:51.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:51.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:51.736 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:52.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:52.076 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:52.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:52.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:52.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:52.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:52.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:52.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:52.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:52.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:52.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:52.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:52.521 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:52.986 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40472 2020-08-24T01:13:52.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40472 '!=' null 2020-08-24T01:13:52.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:52.987 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:52.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:53.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:53.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:53.089 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:53.089 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:53.089 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:53.090 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:53.090 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:53.090 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:53.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:53.443 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:53.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:53.444 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:53.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:53.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:53.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:53.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:53.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:53.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:53.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:53.898 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:53.898 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:54.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27192 2020-08-24T01:13:54.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27192 '!=' null 2020-08-24T01:13:54.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:54.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:54.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:54.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:54.532 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:54.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:54.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:54.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:54.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:54.536 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:54.536 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:54.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:54.888 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:54.889 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:54.889 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:55.343 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:55.344 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:55.344 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:55.344 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:55.344 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:55.344 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:55.345 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:55.346 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:55.346 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:55.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27192 2020-08-24T01:13:55.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27192 '!=' null 2020-08-24T01:13:55.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:55.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:55.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:55.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:55.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:55.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:55.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:55.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:55.893 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:55.894 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:55.894 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:56.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:56.242 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:56.243 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:56.243 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:56.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:56.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:56.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:56.690 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:56.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:56.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:56.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:56.692 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:56.692 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:57.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27192 2020-08-24T01:13:57.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27192 '!=' null 2020-08-24T01:13:57.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:57.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:57.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:57.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:57.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:57.268 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:57.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:57.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:57.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:57.270 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:57.270 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:57.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:57.620 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:57.621 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:57.621 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:58.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:58.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:58.073 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:58.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:58.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:58.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:58.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:58.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:58.077 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:58.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40472 2020-08-24T01:13:58.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40472 '!=' null 2020-08-24T01:13:58.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:58.511 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:58.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:58.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:58.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:58.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:58.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:13:58.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:13:58.963 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:13:58.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:13:58.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:13:59.403 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:13:59.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:13:59.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:13:59.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:13:59.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:13:59.404 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:13:59.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:13:59.405 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:13:59.406 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:13:59.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40472 2020-08-24T01:13:59.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40472 '!=' null 2020-08-24T01:13:59.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:13:59.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:13:59.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:13:59.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:13:59.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:13:59.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:13:59.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:13:59.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:13:59.964 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:13:59.965 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:13:59.965 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:00.298 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:00.300 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:00.301 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:00.301 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:00.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:00.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:00.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:00.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:00.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:00.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:00.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:00.789 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:00.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:01.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27191 2020-08-24T01:14:01.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27191 '!=' null 2020-08-24T01:14:01.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:01.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:01.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:01.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:01.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:01.322 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:01.323 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:01.323 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:01.323 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:01.324 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:01.324 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:01.673 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:01.677 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:01.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:01.678 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:02.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:02.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:02.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:02.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:02.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:02.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:02.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:02.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:02.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:02.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40472 2020-08-24T01:14:02.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40472 '!=' null 2020-08-24T01:14:02.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:02.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:02.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:02.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:02.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:02.694 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:02.694 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:02.694 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:02.695 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:02.695 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:02.695 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:03.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:03.036 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:03.036 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:03.037 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:03.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:03.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:03.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:03.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:03.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:03.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:03.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:03.462 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:03.462 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:03.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40472 2020-08-24T01:14:03.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40472 '!=' null 2020-08-24T01:14:03.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:03.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:03.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:04.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:04.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:04.006 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:04.006 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:04.007 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:04.007 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:04.007 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:04.008 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:04.405 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:04.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:04.407 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:04.408 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:04.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:04.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:04.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:04.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:04.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:04.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:04.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:04.848 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:04.848 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:05.282 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26559 2020-08-24T01:14:05.282 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26559 '!=' null 2020-08-24T01:14:05.282 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:05.283 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:05.283 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:05.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:05.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:05.386 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:05.386 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:05.386 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:05.386 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:05.387 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:05.387 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:05.722 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:05.724 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:05.725 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:05.725 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:06.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:06.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:06.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:06.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:06.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:06.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:06.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:06.176 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:06.177 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:06.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26558 2020-08-24T01:14:06.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26558 '!=' null 2020-08-24T01:14:06.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:06.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:06.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:06.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:06.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:06.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:06.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:06.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:06.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:06.720 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:06.720 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:07.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:07.059 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:07.059 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:07.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:07.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:07.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:07.516 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:07.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:07.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:07.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:07.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:07.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:07.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:08.006 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26558 2020-08-24T01:14:08.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26558 '!=' null 2020-08-24T01:14:08.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:08.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:08.007 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:08.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:08.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:08.111 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:08.111 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:08.124 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:08.125 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:08.126 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:08.127 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:08.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:08.504 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:08.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:08.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:08.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:08.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:08.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:08.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:08.979 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:08.979 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:08.979 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:08.980 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:08.980 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:09.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=40471 2020-08-24T01:14:09.518 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 40471 '!=' null 2020-08-24T01:14:09.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:09.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:09.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:09.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:09.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:09.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:09.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:09.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:09.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:09.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:09.623 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:09.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:09.987 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:09.988 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:09.988 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:10.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:10.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:10.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:10.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:10.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:10.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:10.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:10.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:10.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:10.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20866 2020-08-24T01:14:10.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20866 '!=' null 2020-08-24T01:14:10.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:10.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:10.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:11.004 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:11.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:11.009 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:11.009 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:11.009 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:11.010 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:11.010 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:11.010 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:11.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:11.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:11.364 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:11.364 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:11.808 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:11.808 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:11.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:11.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:11.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:11.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:11.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:11.810 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:11.811 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:12.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20866 2020-08-24T01:14:12.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20866 '!=' null 2020-08-24T01:14:12.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:12.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:12.274 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:12.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:12.374 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:12.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:12.376 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:12.377 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:12.377 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:12.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:12.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:12.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:14:12.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:12.735 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:12.736 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:13.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:14:13.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:14:13.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:14:13.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:14:13.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:14:13.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:14:13.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:14:13.192 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:14:13.193 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:14:13.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20866 2020-08-24T01:14:13.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20866 '!=' null 2020-08-24T01:14:13.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:14:13.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:14:13.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:14:13.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:14:13.738 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:14:13.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:14:13.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:14:13.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:14:13.743 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:14:13.744 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:14:13.744 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:14:14.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:14:14.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:14:14.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:14:14.095 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:14:14.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T01:14:14.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:14:14.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:14:14.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2020-08-24T01:14:14.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:14:14.546 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:14:14.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:14:14.904 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:14:14.904 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:14:14.904 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:14:14.904 INFO:tasks.workunit.client.0.smithi060.stderr:4 2020-08-24T01:14:14.905 INFO:tasks.workunit.client.0.smithi060.stderr:5' 2020-08-24T01:14:14.905 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:14:14.905 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:14:14.906 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:14:15.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836505 2020-08-24T01:14:15.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836505 2020-08-24T01:14:15.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505' 2020-08-24T01:14:15.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:14:15.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:14:15.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672984 2020-08-24T01:14:15.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672984 2020-08-24T01:14:15.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-42949672984' 2020-08-24T01:14:15.250 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:14:15.251 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:14:15.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509463 2020-08-24T01:14:15.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509463 2020-08-24T01:14:15.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509463' 2020-08-24T01:14:15.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:14:15.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:14:15.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345941 2020-08-24T01:14:15.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 85899345941 2020-08-24T01:14:15.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509463 3-85899345941' 2020-08-24T01:14:15.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:14:15.573 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:14:15.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182420 2020-08-24T01:14:15.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 107374182420 2020-08-24T01:14:15.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509463 3-85899345941 4-107374182420' 2020-08-24T01:14:15.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:14:15.725 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-08-24T01:14:15.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018898 2020-08-24T01:14:15.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 128849018898 2020-08-24T01:14:15.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-42949672984 2-64424509463 3-85899345941 4-107374182420 5-128849018898' 2020-08-24T01:14:15.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:14:15.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836505 2020-08-24T01:14:15.883 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:14:15.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:14:15.884 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836505 2020-08-24T01:14:15.885 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:14:15.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836505 2020-08-24T01:14:15.887 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2020-08-24T01:14:15.887 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836505 2020-08-24T01:14:15.889 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:14:16.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836505 -lt 21474836505 2020-08-24T01:14:16.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:14:16.283 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672984 2020-08-24T01:14:16.283 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:14:16.284 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:14:16.287 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672984 2020-08-24T01:14:16.287 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:14:16.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672984 2020-08-24T01:14:16.289 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672984' 2020-08-24T01:14:16.289 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672984 2020-08-24T01:14:16.291 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:14:16.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672984 -lt 42949672984 2020-08-24T01:14:16.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:14:16.684 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509463 2020-08-24T01:14:16.684 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:14:16.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:14:16.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509463 2020-08-24T01:14:16.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:14:16.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509463 2020-08-24T01:14:16.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509463' 2020-08-24T01:14:16.688 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509463 2020-08-24T01:14:16.689 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:14:17.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509463 -lt 64424509463 2020-08-24T01:14:17.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:14:17.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-85899345941 2020-08-24T01:14:17.051 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:14:17.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:14:17.055 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:14:17.055 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345941 2020-08-24T01:14:17.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=85899345941 2020-08-24T01:14:17.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 85899345941' 2020-08-24T01:14:17.058 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 85899345941 2020-08-24T01:14:17.058 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:14:17.448 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 85899345941 -lt 85899345941 2020-08-24T01:14:17.448 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:14:17.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-107374182420 2020-08-24T01:14:17.449 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:14:17.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:14:17.453 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182420 2020-08-24T01:14:17.454 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:14:17.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=107374182420 2020-08-24T01:14:17.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 107374182420' 2020-08-24T01:14:17.456 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 107374182420 2020-08-24T01:14:17.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:14:17.809 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 107374182420 -lt 107374182420 2020-08-24T01:14:17.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:14:17.812 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 5-128849018898 2020-08-24T01:14:17.813 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:14:17.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=5 2020-08-24T01:14:17.816 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018898 2020-08-24T01:14:17.816 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:14:17.817 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.5 seq 128849018898 2020-08-24T01:14:17.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=128849018898 2020-08-24T01:14:17.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.5 seq 128849018898' 2020-08-24T01:14:17.819 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 5 2020-08-24T01:14:18.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 128849018899 -lt 128849018898 2020-08-24T01:14:18.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2020-08-24T01:14:18.197 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2020-08-24T01:14:18.559 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:14:18.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2020-08-24T01:14:18.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2020-08-24T01:14:18.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:14:18.572 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:14:18.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:14:18.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:14:18.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:14:18.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:14:18.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2020-08-24T01:14:18.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-08-24T01:14:18.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-08-24T01:14:18.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:18.895 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:14:18.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-08-24T01:14:18.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:14:18.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-08-24T01:14:18.896 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-08-24T01:14:20.114 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:14:20.114 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_log_t": { 2020-08-24T01:14:20.115 INFO:tasks.workunit.client.0.smithi060.stdout: "head": "34'150", 2020-08-24T01:14:20.115 INFO:tasks.workunit.client.0.smithi060.stdout: "tail": "34'148", 2020-08-24T01:14:20.115 INFO:tasks.workunit.client.0.smithi060.stdout: "log": [ 2020-08-24T01:14:20.115 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.115 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:14:20.116 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-08-24T01:14:20.116 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'149", 2020-08-24T01:14:20.116 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:14:20.116 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4635.0:1", 2020-08-24T01:14:20.116 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:14:20.116 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:13:29.014519+0000", 2020-08-24T01:14:20.117 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:14:20.117 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:14:20.117 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:14:20.117 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:14:20.119 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:14:20.119 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:14:20.120 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.122 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.122 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:14:20.122 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:14:20.122 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:14:20.122 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:14:20.123 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:14:20.123 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.123 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.123 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.123 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.124 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:14:20.124 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-08-24T01:14:20.124 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'150", 2020-08-24T01:14:20.124 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:14:20.124 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4638.0:1", 2020-08-24T01:14:20.125 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:14:20.125 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:13:29.158145+0000", 2020-08-24T01:14:20.125 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:14:20.125 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:14:20.125 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:14:20.126 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:14:20.126 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:14:20.126 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:14:20.126 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.126 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.126 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:14:20.127 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:14:20.127 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:14:20.127 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:14:20.127 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:14:20.127 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.128 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.128 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.128 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:14:20.128 INFO:tasks.workunit.client.0.smithi060.stdout: "dups": [ 2020-08-24T01:14:20.128 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.129 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4611.0:1", 2020-08-24T01:14:20.129 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'141", 2020-08-24T01:14:20.129 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "141", 2020-08-24T01:14:20.129 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.129 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.129 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.130 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4614.0:1", 2020-08-24T01:14:20.130 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'142", 2020-08-24T01:14:20.130 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "142", 2020-08-24T01:14:20.130 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.130 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.131 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.131 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4617.0:1", 2020-08-24T01:14:20.131 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'143", 2020-08-24T01:14:20.131 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "143", 2020-08-24T01:14:20.131 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.132 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.132 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.132 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4620.0:1", 2020-08-24T01:14:20.132 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'144", 2020-08-24T01:14:20.132 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "144", 2020-08-24T01:14:20.132 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.133 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.133 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.133 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4623.0:1", 2020-08-24T01:14:20.133 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'145", 2020-08-24T01:14:20.133 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "145", 2020-08-24T01:14:20.134 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.134 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.134 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.134 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4626.0:1", 2020-08-24T01:14:20.134 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'146", 2020-08-24T01:14:20.135 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "146", 2020-08-24T01:14:20.135 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.136 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.136 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.136 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4629.0:1", 2020-08-24T01:14:20.137 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'147", 2020-08-24T01:14:20.137 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "147", 2020-08-24T01:14:20.137 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.138 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.138 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:14:20.138 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4632.0:1", 2020-08-24T01:14:20.138 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'148", 2020-08-24T01:14:20.138 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "148", 2020-08-24T01:14:20.139 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:14:20.139 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.139 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:14:20.139 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:14:20.139 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_missing_t": { 2020-08-24T01:14:20.140 INFO:tasks.workunit.client.0.smithi060.stdout: "missing": [], 2020-08-24T01:14:20.140 INFO:tasks.workunit.client.0.smithi060.stdout: "may_include_deletes": true 2020-08-24T01:14:20.140 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:14:20.140 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:14:20.141 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-08-24T01:14:20.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2020-08-24T01:14:20.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2020-08-24T01:14:20.142 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-08-24T01:14:20.142 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=8 2020-08-24T01:14:20.143 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 8 '!=' 8 ']' 2020-08-24T01:14:20.143 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-08-24T01:14:20.143 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to max 2 2020-08-24T01:14:20.143 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to copy log version 34'150 2020-08-24T01:14:20.143 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to copy log version 34'149 2020-08-24T01:14:20.144 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after earliest_dup_version 141 2020-08-24T01:14:20.144 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-08-24T01:14:20.144 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-08-24T01:14:20.144 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-08-24T01:14:20.144 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-08-24T01:14:20.145 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-08-24T01:14:20.145 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-08-24T01:14:20.145 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-08-24T01:14:20.145 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.099+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-08-24T01:14:20.146 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to max 2 2020-08-24T01:14:20.146 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to copy log version 34'150 2020-08-24T01:14:20.146 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to copy log version 34'149 2020-08-24T01:14:20.146 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after earliest_dup_version 141 2020-08-24T01:14:20.147 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-08-24T01:14:20.147 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-08-24T01:14:20.147 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-08-24T01:14:20.147 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-08-24T01:14:20.147 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-08-24T01:14:20.148 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-08-24T01:14:20.148 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-08-24T01:14:20.148 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-08-24T01:14:20.148 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to max 2 2020-08-24T01:14:20.148 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to copy log version 34'150 2020-08-24T01:14:20.149 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to copy log version 34'149 2020-08-24T01:14:20.149 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after earliest_dup_version 141 2020-08-24T01:14:20.149 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-08-24T01:14:20.149 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-08-24T01:14:20.157 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-08-24T01:14:20.160 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-08-24T01:14:20.160 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-08-24T01:14:20.161 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-08-24T01:14:20.161 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-08-24T01:14:20.161 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:13:32.103+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-08-24T01:14:20.161 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_after v 34'150 2020-08-24T01:14:20.161 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after earliest_dup_version 141 2020-08-24T01:14:20.162 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'141 2020-08-24T01:14:20.162 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'142 2020-08-24T01:14:20.162 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'143 2020-08-24T01:14:20.162 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'144 2020-08-24T01:14:20.163 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'145 2020-08-24T01:14:20.163 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'146 2020-08-24T01:14:20.163 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'147 2020-08-24T01:14:20.163 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'148 2020-08-24T01:14:20.163 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'149 2020-08-24T01:14:20.163 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:14:09.666+0000 7ff933490700 20 copy_up_to/copy_after copy dup from log version 34'150 2020-08-24T01:14:20.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-08-24T01:14:20.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2020-08-24T01:14:20.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-08-24T01:14:20.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:14:20.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T01:14:20.175 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:14:20.175 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:14:20.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:14:20.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:14:20.176 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:14:20.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:14:20.178 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:14:20.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:14:20.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:14:20.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:14:20.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:14:20.181 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:14:20.181 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:14:20.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:14:20.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:14:20.182 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:14:20.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:14:20.185 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:14:20.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T01:14:20.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:14:20.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:20.206 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:20.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T01:14:20.207 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:14:20.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:14:20.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-08-24T01:14:20.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-08-24T01:14:20.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.208 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-08-24T01:14:20.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:14:20.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T01:14:20.209 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:14:20.209 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:14:20.210 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:14:20.210 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:14:20.210 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:14:20.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:14:20.211 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:14:20.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:14:20.212 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:14:20.213 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:14:20.214 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:14:20.214 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:14:20.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:14:20.215 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:14:20.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:14:20.216 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:14:20.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:14:20.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:14:20.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T01:14:20.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:14:20.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:20.220 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:20.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T01:14:20.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:14:20.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:14:20.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-08-24T01:14:20.224 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:14:20.224 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:20.224 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:20.225 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71396 2020-08-24T01:14:20.231 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:14:20.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-08-24T01:14:20.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:14:20.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-08-24T01:14:20.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-08-24T01:14:20.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:115: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-08-24T01:14:20.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:20.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-08-24T01:14:20.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-08-24T01:14:20.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-08-24T01:14:20.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-08-24T01:14:20.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-08-24T01:14:20.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-08-24T01:14:20.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:14:20.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:14:20.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:14:20.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-08-24T01:14:20.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.559 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:14:20.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:20.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:20.561 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:20.561 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:20.562 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:20.562 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:20.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:14:20.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:14:20.745 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:14:20.745 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:14:20.746 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:14:20.746 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:14:20.748 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:14:20.748 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:14:20.748 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:14:20.748 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:14:20.749 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:14:20.749 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:20.749 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:20.749 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:14:20.750 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:14:20.750 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get fsid 2020-08-24T01:14:20.848 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:14:20.848 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:14:20.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:14:20.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:14:20.853 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:14:20.854 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:14:20.854 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:14:20.854 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:14:20.854 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:14:20.855 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:20.855 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:20.855 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:14:20.855 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:14:20.855 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get mon_host 2020-08-24T01:14:20.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-08-24T01:14:20.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:20.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:14:20.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:14:20.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:14:20.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-08-24T01:14:20.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:14:21.508 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:14:21.509 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:21.509 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:21.510 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:21.510 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:21.510 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:21.511 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:21.511 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:14:21.512 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:14:21.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-08-24T01:14:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:21.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:21.542 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-08-24T01:14:21.544 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-08-24T01:14:21.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:14:21.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-08-24T01:14:21.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:21.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:14:21.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:14:21.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:14:21.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-08-24T01:14:21.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:14:21.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:14:21.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:14:21.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:14:21.548 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-08-24T01:14:21.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-08-24T01:14:21.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:14:21.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:21.549 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:14:21.549 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:14:21.550 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:21.550 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:21.552 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:21.552 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:21.552 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:21.552 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:21.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:21.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:14:21.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:14:21.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:14:21.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:14:21.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:14:21.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:14:21.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:14:21.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:21.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:14:21.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:14:21.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-08-24T01:14:21.558 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:14:21.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d3db7e96-e4b9-4e90-9b1f-cb2cb2ca4d17 2020-08-24T01:14:21.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 d3db7e96-e4b9-4e90-9b1f-cb2cb2ca4d17' 2020-08-24T01:14:21.559 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:14:21.559 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 d3db7e96-e4b9-4e90-9b1f-cb2cb2ca4d17 2020-08-24T01:14:21.572 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBtFENfpeTtIRAAOLW76J/7xW2ZIhQj9n7NAw== 2020-08-24T01:14:21.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBtFENfpeTtIRAAOLW76J/7xW2ZIhQj9n7NAw=="}' 2020-08-24T01:14:21.573 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d3db7e96-e4b9-4e90-9b1f-cb2cb2ca4d17 -i td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:14:21.905 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:14:21.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:14:21.918 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBtFENfpeTtIRAAOLW76J/7xW2ZIhQj9n7NAw== --osd-uuid d3db7e96-e4b9-4e90-9b1f-cb2cb2ca4d17 2020-08-24T01:14:21.947 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:21.942+0000 7fd5d057ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:21.950 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:21.946+0000 7fd5d057ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:21.951 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:21.946+0000 7fd5d057ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:21.952 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:21.950+0000 7fd5d057ed80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:14:21.952 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:21.950+0000 7fd5d057ed80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:14:21.952 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:21.950+0000 7fd5d057ed80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:14:21.998 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:21.994+0000 7fd5d057ed80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-08-24T01:14:22.352 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:22.350+0000 7fd5d057ed80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:14:25.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-08-24T01:14:25.072 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:14:25.074 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:14:25.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:14:25.074 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:14:25.321 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:14:25.321 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:14:25.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:14:25.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:25.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:14:25.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:14:25.350 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:25.346+0000 7fdcfb42cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:25.352 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:25.350+0000 7fdcfb42cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:25.353 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:25.350+0000 7fdcfb42cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:25.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:14:25.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:14:25.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:14:25.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:14:25.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:14:25.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:25.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:14:25.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:25.665 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:14:25.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:14:25.885 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:25.882+0000 7fdcfb42cd80 -1 Falling back to public interface 2020-08-24T01:14:26.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:14:26.633 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:26.626+0000 7fdcfb42cd80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:14:27.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:14:27.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:27.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:14:27.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:27.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:14:27.027 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:14:27.432 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/81070,v1:127.0.0.1:6803/81070] [v2:127.0.0.1:6804/81070,v1:127.0.0.1:6805/81070] exists,up d3db7e96-e4b9-4e90-9b1f-cb2cb2ca4d17 2020-08-24T01:14:27.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:14:27.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:14:27.433 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:14:27.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:14:27.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-08-24T01:14:27.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:27.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:14:27.434 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:14:27.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:14:27.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-08-24T01:14:27.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:14:27.435 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:14:27.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:14:27.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:14:27.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-08-24T01:14:27.436 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-08-24T01:14:27.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:14:27.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:27.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:14:27.437 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:14:27.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:27.438 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:27.438 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:27.438 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:27.438 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:27.441 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:27.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:27.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:14:27.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:14:27.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:14:27.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:14:27.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:14:27.442 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:14:27.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:14:27.449 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:27.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:14:27.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:14:27.450 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-08-24T01:14:27.450 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:14:27.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7eb01dd5-b47c-42e4-9b65-e248d587cee6 2020-08-24T01:14:27.451 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 7eb01dd5-b47c-42e4-9b65-e248d587cee6' 2020-08-24T01:14:27.451 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:14:27.451 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 7eb01dd5-b47c-42e4-9b65-e248d587cee6 2020-08-24T01:14:27.456 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBzFENfCjUeGxAAzQ9jXdgyY4NtgRzHD6KKJA== 2020-08-24T01:14:27.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBzFENfCjUeGxAAzQ9jXdgyY4NtgRzHD6KKJA=="}' 2020-08-24T01:14:27.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7eb01dd5-b47c-42e4-9b65-e248d587cee6 -i td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:14:27.946 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:14:27.958 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:14:27.960 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBzFENfCjUeGxAAzQ9jXdgyY4NtgRzHD6KKJA== --osd-uuid 7eb01dd5-b47c-42e4-9b65-e248d587cee6 2020-08-24T01:14:27.986 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:27.982+0000 7ffbbfb23d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:28.005 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:28.002+0000 7ffbbfb23d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:28.007 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:28.002+0000 7ffbbfb23d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:28.008 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:28.006+0000 7ffbbfb23d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:14:28.008 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:28.006+0000 7ffbbfb23d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:14:28.008 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:28.006+0000 7ffbbfb23d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:14:28.100 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:28.098+0000 7ffbbfb23d80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-08-24T01:14:28.451 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:28.446+0000 7ffbbfb23d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:14:31.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-08-24T01:14:31.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:14:31.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:14:31.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:14:31.202 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:14:31.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:14:31.670 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:14:31.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:31.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:14:31.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:14:31.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:14:31.700 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:31.698+0000 7efc0d630d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:31.707 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:31.698+0000 7efc0d630d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:31.707 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:31.702+0000 7efc0d630d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:32.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:14:32.019 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:14:32.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:14:32.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:14:32.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:14:32.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:32.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:14:32.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:32.021 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:14:32.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:14:32.242 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:32.238+0000 7efc0d630d80 -1 Falling back to public interface 2020-08-24T01:14:32.385 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:14:32.987 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:32.982+0000 7efc0d630d80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:14:33.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:14:33.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:33.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:14:33.388 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:14:33.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:33.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:14:33.766 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/81425,v1:127.0.0.1:6811/81425] [v2:127.0.0.1:6812/81425,v1:127.0.0.1:6813/81425] exists,up 7eb01dd5-b47c-42e4-9b65-e248d587cee6 2020-08-24T01:14:33.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:14:33.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:14:33.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:14:33.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:14:33.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-08-24T01:14:33.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:33.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:14:33.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T01:14:33.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:14:33.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-08-24T01:14:33.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:14:33.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:14:33.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:14:33.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:14:33.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-08-24T01:14:33.769 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-08-24T01:14:33.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:14:33.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:33.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:14:33.772 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:14:33.773 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:33.773 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:33.773 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:33.773 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:33.773 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:33.774 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:33.774 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:33.774 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:14:33.774 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:14:33.774 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:14:33.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:14:33.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:14:33.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:14:33.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:14:33.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:33.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:14:33.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:14:33.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-08-24T01:14:33.777 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:14:33.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3211cd19-8603-4763-a346-a950f9271069 2020-08-24T01:14:33.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 3211cd19-8603-4763-a346-a950f9271069' 2020-08-24T01:14:33.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:14:33.778 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 3211cd19-8603-4763-a346-a950f9271069 2020-08-24T01:14:33.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQB5FENfaEbuLhAAzx4pr47WvnHTtUYXprijeg== 2020-08-24T01:14:33.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQB5FENfaEbuLhAAzx4pr47WvnHTtUYXprijeg=="}' 2020-08-24T01:14:33.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3211cd19-8603-4763-a346-a950f9271069 -i td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:14:34.313 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:14:34.324 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:14:34.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB5FENfaEbuLhAAzx4pr47WvnHTtUYXprijeg== --osd-uuid 3211cd19-8603-4763-a346-a950f9271069 2020-08-24T01:14:34.351 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.346+0000 7f05678a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:34.353 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.350+0000 7f05678a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:34.358 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.350+0000 7f05678a3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:34.358 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.354+0000 7f05678a3d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:14:34.358 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.354+0000 7f05678a3d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:14:34.359 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.354+0000 7f05678a3d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:14:34.424 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.422+0000 7f05678a3d80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-08-24T01:14:34.835 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:34.830+0000 7f05678a3d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:14:37.505 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-08-24T01:14:37.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:14:37.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T01:14:37.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:14:37.507 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T01:14:37.967 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:14:37.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T01:14:37.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:37.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:14:37.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:14:37.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:14:37.997 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:37.994+0000 7ff11b1f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:37.999 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:37.994+0000 7ff11b1f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:38.000 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:37.998+0000 7ff11b1f3d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:38.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T01:14:38.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:14:38.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:14:38.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:14:38.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:14:38.325 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:38.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:14:38.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:38.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:14:38.327 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:14:38.537 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:38.534+0000 7ff11b1f3d80 -1 Falling back to public interface 2020-08-24T01:14:38.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:14:39.308 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:39.306+0000 7ff11b1f3d80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T01:14:39.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:14:39.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:39.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:14:39.694 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:14:39.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:39.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:14:40.106 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/81780,v1:127.0.0.1:6819/81780] [v2:127.0.0.1:6820/81780,v1:127.0.0.1:6821/81780] exists,up 3211cd19-8603-4763-a346-a950f9271069 2020-08-24T01:14:40.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:14:40.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:14:40.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:14:40.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:14:40.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-08-24T01:14:40.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:40.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:14:40.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-08-24T01:14:40.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:14:40.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-08-24T01:14:40.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:14:40.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:14:40.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:14:40.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:14:40.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-08-24T01:14:40.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-08-24T01:14:40.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:14:40.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:40.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:14:40.111 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:14:40.111 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:40.111 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:40.111 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:40.111 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:40.112 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:40.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:40.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:40.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:14:40.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:14:40.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:14:40.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:14:40.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:14:40.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:14:40.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:14:40.115 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:40.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:14:40.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:14:40.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-08-24T01:14:40.116 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:14:40.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6071787a-f19e-46e8-959e-43a62824346d 2020-08-24T01:14:40.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 6071787a-f19e-46e8-959e-43a62824346d' 2020-08-24T01:14:40.117 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:14:40.117 INFO:tasks.workunit.client.0.smithi060.stdout:add osd3 6071787a-f19e-46e8-959e-43a62824346d 2020-08-24T01:14:40.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCAFENfObafBxAA0A3F60e5QQ7u/qxJel5CaQ== 2020-08-24T01:14:40.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCAFENfObafBxAA0A3F60e5QQ7u/qxJel5CaQ=="}' 2020-08-24T01:14:40.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6071787a-f19e-46e8-959e-43a62824346d -i td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:14:40.691 INFO:tasks.workunit.client.0.smithi060.stdout:3 2020-08-24T01:14:40.700 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:14:40.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCAFENfObafBxAA0A3F60e5QQ7u/qxJel5CaQ== --osd-uuid 6071787a-f19e-46e8-959e-43a62824346d 2020-08-24T01:14:40.728 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:40.726+0000 7f7655aded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:40.731 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:40.726+0000 7f7655aded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:40.732 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:40.730+0000 7f7655aded80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:40.733 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:40.730+0000 7f7655aded80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:14:40.734 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:40.730+0000 7f7655aded80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:14:40.734 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:40.730+0000 7f7655aded80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:14:40.780 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:40.774+0000 7f7655aded80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-08-24T01:14:41.214 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:41.210+0000 7f7655aded80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:14:43.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-08-24T01:14:43.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:14:43.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-08-24T01:14:43.935 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:14:43.935 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd3 key to auth repository 2020-08-24T01:14:44.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-08-24T01:14:44.411 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.3 2020-08-24T01:14:44.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:44.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:14:44.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:14:44.413 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:14:44.442 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:44.438+0000 7f6213e6ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:44.443 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:44.438+0000 7f6213e6ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:44.444 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:44.442+0000 7f6213e6ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:44.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-08-24T01:14:44.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:14:44.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-08-24T01:14:44.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:14:44.765 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:14:44.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:44.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:14:44.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:44.766 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:14:44.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:14:44.985 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:44.982+0000 7f6213e6ad80 -1 Falling back to public interface 2020-08-24T01:14:45.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:14:45.808 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:45.802+0000 7f6213e6ad80 -1 osd.3 0 log_to_monitors {default=true} 2020-08-24T01:14:46.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:14:46.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:46.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:14:46.165 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:14:46.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:46.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:14:46.554 INFO:tasks.workunit.client.0.smithi060.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/82135,v1:127.0.0.1:6827/82135] [v2:127.0.0.1:6828/82135,v1:127.0.0.1:6829/82135] exists,up 6071787a-f19e-46e8-959e-43a62824346d 2020-08-24T01:14:46.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:14:46.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:14:46.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:14:46.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:14:46.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-08-24T01:14:46.555 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:46.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:14:46.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-08-24T01:14:46.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:14:46.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-08-24T01:14:46.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:14:46.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:14:46.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:14:46.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:14:46.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-08-24T01:14:46.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-08-24T01:14:46.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:14:46.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:46.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:14:46.558 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:14:46.558 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:46.559 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:46.561 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:46.561 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:46.561 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:46.561 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:46.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:46.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:14:46.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:14:46.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:14:46.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:14:46.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:14:46.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:14:46.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:14:46.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:46.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:14:46.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:14:46.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-08-24T01:14:46.564 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:14:46.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=99de6a4f-a024-4199-9e2e-004d21000434 2020-08-24T01:14:46.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 99de6a4f-a024-4199-9e2e-004d21000434' 2020-08-24T01:14:46.564 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:14:46.565 INFO:tasks.workunit.client.0.smithi060.stdout:add osd4 99de6a4f-a024-4199-9e2e-004d21000434 2020-08-24T01:14:46.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCGFENfZc00IhAAzIUbjvrAGz/Yb40LNigF5A== 2020-08-24T01:14:46.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCGFENfZc00IhAAzIUbjvrAGz/Yb40LNigF5A=="}' 2020-08-24T01:14:46.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 99de6a4f-a024-4199-9e2e-004d21000434 -i td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:14:47.115 INFO:tasks.workunit.client.0.smithi060.stdout:4 2020-08-24T01:14:47.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:14:47.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCGFENfZc00IhAAzIUbjvrAGz/Yb40LNigF5A== --osd-uuid 99de6a4f-a024-4199-9e2e-004d21000434 2020-08-24T01:14:47.159 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.154+0000 7f05b7bfbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:47.205 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.202+0000 7f05b7bfbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:47.213 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.202+0000 7f05b7bfbd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:47.214 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.206+0000 7f05b7bfbd80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:14:47.214 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.206+0000 7f05b7bfbd80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:14:47.214 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.206+0000 7f05b7bfbd80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:14:47.272 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.270+0000 7f05b7bfbd80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-08-24T01:14:47.780 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:47.778+0000 7f05b7bfbd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:14:50.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-08-24T01:14:50.800 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:14:50.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-08-24T01:14:50.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:14:50.803 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd4 key to auth repository 2020-08-24T01:14:51.269 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-08-24T01:14:51.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:51.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:14:51.270 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.4 2020-08-24T01:14:51.271 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:14:51.271 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:14:51.306 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:51.294+0000 7f634de6dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:51.306 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:51.298+0000 7f634de6dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:51.307 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:51.298+0000 7f634de6dd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:51.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-08-24T01:14:51.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:14:51.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-08-24T01:14:51.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:14:51.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:14:51.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:51.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:14:51.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:51.629 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:14:51.637 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:14:51.842 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:51.838+0000 7f634de6dd80 -1 Falling back to public interface 2020-08-24T01:14:51.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:14:52.639 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:52.630+0000 7f634de6dd80 -1 osd.4 0 log_to_monitors {default=true} 2020-08-24T01:14:52.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:14:52.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:52.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:14:52.970 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:14:52.970 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:52.971 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:14:53.347 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:14:54.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:14:54.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:54.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:14:54.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:54.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:14:54.352 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:14:54.723 INFO:tasks.workunit.client.0.smithi060.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/82490,v1:127.0.0.1:6835/82490] [v2:127.0.0.1:6836/82490,v1:127.0.0.1:6837/82490] exists,up 99de6a4f-a024-4199-9e2e-004d21000434 2020-08-24T01:14:54.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:14:54.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:14:54.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:14:54.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:14:54.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-08-24T01:14:54.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:14:54.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:14:54.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-08-24T01:14:54.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:14:54.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-08-24T01:14:54.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:14:54.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:14:54.730 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:14:54.731 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:14:54.731 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-08-24T01:14:54.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-08-24T01:14:54.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:14:54.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-08-24T01:14:54.738 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:14:54.738 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:14:54.738 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:14:54.739 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:14:54.739 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:14:54.739 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:14:54.739 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:14:54.740 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:54.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:14:54.744 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:14:54.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:14:54.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:14:54.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:14:54.745 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:14:54.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:14:54.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:14:54.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:54.752 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:14:54.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:14:54.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-08-24T01:14:54.753 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:14:54.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=83ec693f-6b90-4f34-b548-287b2f477af1 2020-08-24T01:14:54.754 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 83ec693f-6b90-4f34-b548-287b2f477af1' 2020-08-24T01:14:54.754 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:14:54.754 INFO:tasks.workunit.client.0.smithi060.stdout:add osd5 83ec693f-6b90-4f34-b548-287b2f477af1 2020-08-24T01:14:54.755 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCOFENfPfFxLBAA+qnaEL8/WyLxUq0Jpjkhww== 2020-08-24T01:14:54.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCOFENfPfFxLBAA+qnaEL8/WyLxUq0Jpjkhww=="}' 2020-08-24T01:14:54.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 83ec693f-6b90-4f34-b548-287b2f477af1 -i td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:14:55.298 INFO:tasks.workunit.client.0.smithi060.stdout:5 2020-08-24T01:14:55.321 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:14:55.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCOFENfPfFxLBAA+qnaEL8/WyLxUq0Jpjkhww== --osd-uuid 83ec693f-6b90-4f34-b548-287b2f477af1 2020-08-24T01:14:55.343 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.338+0000 7fa0bb368d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:55.360 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.358+0000 7fa0bb368d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:55.362 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.358+0000 7fa0bb368d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:55.362 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.358+0000 7fa0bb368d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:14:55.363 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.358+0000 7fa0bb368d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:14:55.363 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.358+0000 7fa0bb368d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:14:55.428 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.422+0000 7fa0bb368d80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-08-24T01:14:55.901 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:55.898+0000 7fa0bb368d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:14:58.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-08-24T01:14:58.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:14:58.864 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-08-24T01:14:58.865 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:14:58.867 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd5 key to auth repository 2020-08-24T01:14:59.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-08-24T01:14:59.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:14:59.351 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.5 2020-08-24T01:14:59.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:14:59.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:14:59.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:14:59.377 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:59.374+0000 7f5cb313cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:59.379 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:59.374+0000 7f5cb313cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:59.380 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:59.378+0000 7f5cb313cd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:14:59.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-08-24T01:14:59.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:14:59.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-08-24T01:14:59.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:14:59.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:14:59.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:14:59.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:14:59.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:14:59.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:14:59.716 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:14:59.926 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:14:59.922+0000 7f5cb313cd80 -1 Falling back to public interface 2020-08-24T01:15:00.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:15:00.710 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:15:00.706+0000 7f5cb313cd80 -1 osd.5 0 log_to_monitors {default=true} 2020-08-24T01:15:01.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:15:01.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:15:01.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:15:01.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:15:01.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:15:01.050 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:15:01.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:15:02.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:15:02.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:15:02.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:15:02.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:15:02.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:15:02.427 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:15:02.801 INFO:tasks.workunit.client.0.smithi060.stdout:osd.5 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/82873,v1:127.0.0.1:6843/82873] [v2:127.0.0.1:6844/82873,v1:127.0.0.1:6845/82873] exists,up 83ec693f-6b90-4f34-b548-287b2f477af1 2020-08-24T01:15:02.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:15:02.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:15:02.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:15:02.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-08-24T01:15:02.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-08-24T01:15:03.349 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T01:15:03.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:15:04.362 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-08-24T01:15:04.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:04.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-08-24T01:15:04.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:04.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-08-24T01:15:04.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:04.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-08-24T01:15:04.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:04.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-08-24T01:15:05.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:05.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-08-24T01:15:05.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:05.191 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-08-24T01:15:05.333 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:05.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-08-24T01:15:05.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:05.493 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-08-24T01:15:05.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:05.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-08-24T01:15:05.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:05.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-08-24T01:15:05.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:05.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-08-24T01:15:06.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:06.061 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-08-24T01:15:06.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:06.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-08-24T01:15:06.338 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:06.339 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-08-24T01:15:06.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:06.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-08-24T01:15:06.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:06.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-08-24T01:15:06.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:06.801 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-08-24T01:15:06.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:06.970 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-08-24T01:15:07.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:07.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-08-24T01:15:07.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:07.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-08-24T01:15:07.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:07.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-08-24T01:15:07.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:07.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-08-24T01:15:07.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:07.630 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-08-24T01:15:07.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:07.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-08-24T01:15:07.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:07.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-08-24T01:15:08.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:08.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-08-24T01:15:08.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:08.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-08-24T01:15:08.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:08.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-08-24T01:15:08.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:08.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-08-24T01:15:08.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:08.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-08-24T01:15:08.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:08.778 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-08-24T01:15:08.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:08.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-08-24T01:15:09.066 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:09.067 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-08-24T01:15:09.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:09.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-08-24T01:15:09.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:09.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-08-24T01:15:09.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:09.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-08-24T01:15:09.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:09.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-08-24T01:15:09.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:09.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-08-24T01:15:09.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:09.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-08-24T01:15:10.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:10.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-08-24T01:15:10.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:10.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-08-24T01:15:10.346 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:10.347 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-08-24T01:15:10.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:10.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-08-24T01:15:10.635 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:10.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-08-24T01:15:10.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:10.771 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-08-24T01:15:10.912 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:10.913 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-08-24T01:15:11.071 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:11.071 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-08-24T01:15:11.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:11.223 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-08-24T01:15:11.343 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:11.343 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-08-24T01:15:11.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:11.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-08-24T01:15:11.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:11.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-08-24T01:15:11.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:11.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-08-24T01:15:11.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:11.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-08-24T01:15:12.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:12.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-08-24T01:15:12.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:12.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-08-24T01:15:12.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:12.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-08-24T01:15:12.490 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:12.491 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-08-24T01:15:12.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:12.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-08-24T01:15:12.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:12.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-08-24T01:15:12.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:12.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-08-24T01:15:13.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:13.031 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-08-24T01:15:13.172 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:13.173 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-08-24T01:15:13.309 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:13.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-08-24T01:15:13.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:13.441 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-08-24T01:15:13.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:13.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-08-24T01:15:13.735 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:13.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-08-24T01:15:13.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:13.884 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-08-24T01:15:14.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:14.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-08-24T01:15:14.145 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:14.146 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-08-24T01:15:14.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:14.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-08-24T01:15:14.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:14.430 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-08-24T01:15:14.574 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:14.575 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-08-24T01:15:14.722 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:14.722 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-08-24T01:15:14.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:14.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-08-24T01:15:15.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:15.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-08-24T01:15:15.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:15.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-08-24T01:15:15.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:15.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-08-24T01:15:15.443 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:15.444 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-08-24T01:15:15.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:15.570 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-08-24T01:15:15.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:15.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-08-24T01:15:15.890 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:15.891 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-08-24T01:15:16.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-08-24T01:15:16.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-08-24T01:15:16.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-08-24T01:15:16.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.458 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-08-24T01:15:16.599 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-08-24T01:15:16.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.734 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-08-24T01:15:16.866 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-08-24T01:15:16.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:16.989 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-08-24T01:15:17.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:17.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-08-24T01:15:17.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:17.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-08-24T01:15:17.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:17.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-08-24T01:15:17.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:17.522 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-08-24T01:15:17.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:17.669 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-08-24T01:15:17.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:17.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-08-24T01:15:17.956 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:17.956 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-08-24T01:15:18.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:18.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-08-24T01:15:18.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:18.220 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-08-24T01:15:18.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:18.352 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-08-24T01:15:18.514 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:18.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-08-24T01:15:18.636 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:18.637 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-08-24T01:15:18.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:18.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-08-24T01:15:18.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:18.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-08-24T01:15:19.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:19.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-08-24T01:15:19.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:19.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-08-24T01:15:19.389 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:19.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-08-24T01:15:19.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:19.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-08-24T01:15:19.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:19.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-08-24T01:15:19.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:19.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-08-24T01:15:19.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:19.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-08-24T01:15:20.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:20.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-08-24T01:15:20.238 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:20.239 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-08-24T01:15:20.378 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:20.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-08-24T01:15:20.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:20.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-08-24T01:15:20.662 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:20.662 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-08-24T01:15:20.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:20.906 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-08-24T01:15:21.042 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:21.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-08-24T01:15:21.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:21.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-08-24T01:15:21.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:21.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-08-24T01:15:21.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:21.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-08-24T01:15:21.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:21.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-08-24T01:15:21.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:21.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-08-24T01:15:21.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:21.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-08-24T01:15:22.070 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:22.070 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-08-24T01:15:22.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:22.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-08-24T01:15:22.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:22.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-08-24T01:15:22.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:22.506 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-08-24T01:15:22.648 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:22.648 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-08-24T01:15:22.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:22.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-08-24T01:15:22.915 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:22.915 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-08-24T01:15:23.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:23.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-08-24T01:15:23.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:23.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-08-24T01:15:23.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:23.351 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-08-24T01:15:23.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:23.468 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-08-24T01:15:23.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:23.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-08-24T01:15:23.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:23.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-08-24T01:15:23.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:23.939 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-08-24T01:15:24.081 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:24.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-08-24T01:15:24.230 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:24.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-08-24T01:15:24.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:24.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-08-24T01:15:24.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:24.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-08-24T01:15:24.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:24.667 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-08-24T01:15:24.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:24.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-08-24T01:15:24.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:24.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-08-24T01:15:25.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:25.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-08-24T01:15:25.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:25.205 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-08-24T01:15:25.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:25.338 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-08-24T01:15:25.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:25.471 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-08-24T01:15:25.595 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:25.596 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-08-24T01:15:25.740 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:15:25.741 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-08-24T01:15:25.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-08-24T01:15:25.899 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-08-24T01:15:26.250 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:15:26.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-08-24T01:15:26.783 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-08-24T01:15:26.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-08-24T01:15:26.792 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-08-24T01:15:27.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-08-24T01:15:27.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:15:27.798 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:15:27.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:15:27.799 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:15:27.799 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:15:27.799 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:15:27.799 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:15:27.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:15:27.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:15:27.800 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:15:28.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:15:28.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:15:28.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:15:28.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:15:28.001 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:15:28.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:15:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:15:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:15:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:15:28.370 INFO:tasks.workunit.client.0.smithi060.stderr:4 2020-08-24T01:15:28.370 INFO:tasks.workunit.client.0.smithi060.stderr:5' 2020-08-24T01:15:28.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:15:28.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:15:28.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:15:28.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836495 2020-08-24T01:15:28.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836495 2020-08-24T01:15:28.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495' 2020-08-24T01:15:28.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:15:28.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:15:28.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705677 2020-08-24T01:15:28.683 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705677 2020-08-24T01:15:28.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-38654705677' 2020-08-24T01:15:28.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:15:28.685 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:15:28.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542157 2020-08-24T01:15:28.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542157 2020-08-24T01:15:28.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-38654705677 2-60129542157' 2020-08-24T01:15:28.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:15:28.836 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:15:28.993 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378635 2020-08-24T01:15:28.994 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378635 2020-08-24T01:15:28.994 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-38654705677 2-60129542157 3-81604378635' 2020-08-24T01:15:28.994 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:15:28.995 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:15:29.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215114 2020-08-24T01:15:29.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 103079215114 2020-08-24T01:15:29.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-38654705677 2-60129542157 3-81604378635 4-103079215114' 2020-08-24T01:15:29.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:15:29.203 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-08-24T01:15:29.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051592 2020-08-24T01:15:29.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 124554051592 2020-08-24T01:15:29.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836495 1-38654705677 2-60129542157 3-81604378635 4-103079215114 5-124554051592' 2020-08-24T01:15:29.355 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:15:29.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836495 2020-08-24T01:15:29.356 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:15:29.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:15:29.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836495 2020-08-24T01:15:29.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:15:29.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836495 2020-08-24T01:15:29.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836495' 2020-08-24T01:15:29.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:15:29.364 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836495 2020-08-24T01:15:29.775 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836495 -lt 21474836495 2020-08-24T01:15:29.776 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:15:29.776 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705677 2020-08-24T01:15:29.776 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:15:29.777 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:15:29.777 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705677 2020-08-24T01:15:29.778 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:15:29.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705677 2020-08-24T01:15:29.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705677' 2020-08-24T01:15:29.780 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:15:29.780 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705677 2020-08-24T01:15:30.162 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705677 -lt 38654705677 2020-08-24T01:15:30.821 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:15:30.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542157 2020-08-24T01:15:30.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:15:30.824 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:15:30.824 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542157 2020-08-24T01:15:30.825 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:15:30.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542157 2020-08-24T01:15:30.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2020-08-24T01:15:30.825 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:15:30.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542157 -lt 60129542157 2020-08-24T01:15:30.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:15:30.826 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378635 2020-08-24T01:15:30.826 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:15:30.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:15:30.826 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378635 2020-08-24T01:15:30.827 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:15:30.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378635 2020-08-24T01:15:30.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378635' 2020-08-24T01:15:30.827 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:15:30.828 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542157 2020-08-24T01:15:30.828 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378635 2020-08-24T01:15:30.981 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378635 -lt 81604378635 2020-08-24T01:15:30.981 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:15:30.981 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-103079215114 2020-08-24T01:15:30.982 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:15:30.983 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:15:30.984 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215114 2020-08-24T01:15:30.984 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:15:30.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=103079215114 2020-08-24T01:15:30.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 103079215114' 2020-08-24T01:15:30.986 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 103079215114 2020-08-24T01:15:30.986 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:15:31.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 103079215114 -lt 103079215114 2020-08-24T01:15:31.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:15:31.420 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 5-124554051592 2020-08-24T01:15:31.420 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:15:31.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=5 2020-08-24T01:15:31.424 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:15:31.425 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051592 2020-08-24T01:15:31.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=124554051592 2020-08-24T01:15:31.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.5 seq 124554051592' 2020-08-24T01:15:31.425 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 5 2020-08-24T01:15:31.426 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.5 seq 124554051592 2020-08-24T01:15:31.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 124554051592 -lt 124554051592 2020-08-24T01:15:31.780 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:15:31.780 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:31.780 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:32.227 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:15:32.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:32.228 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:32.228 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:32.229 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:32.229 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:32.229 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:32.229 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:32.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:32.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:32.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:32.560 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:33.013 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:33.013 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T01:15:33.013 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:15:33.014 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T01:15:33.014 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:33.014 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:33.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:33.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:33.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:33.112 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:33.113 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:33.113 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:33.113 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:33.113 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:33.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:33.485 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:33.485 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:33.486 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:33.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:33.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:33.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:33.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:33.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:33.925 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:33.926 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:33.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:33.926 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:34.372 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21500 2020-08-24T01:15:34.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21500 '!=' null 2020-08-24T01:15:34.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:34.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:34.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:34.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:34.474 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:34.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:34.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:34.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:34.475 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:34.476 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:34.476 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:34.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:34.826 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:34.826 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:34.827 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:35.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:35.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:35.273 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:35.274 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:35.274 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:35.274 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:35.274 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:35.275 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:35.275 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:35.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=16274 2020-08-24T01:15:35.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 16274 '!=' null 2020-08-24T01:15:35.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:35.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:35.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:35.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:35.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:35.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:35.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:35.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:35.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:35.805 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:35.806 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:36.147 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:36.148 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:36.148 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:36.148 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:36.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:36.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:36.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:36.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:36.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:36.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:36.589 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:36.589 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:36.590 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:37.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=16274 2020-08-24T01:15:37.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 16274 '!=' null 2020-08-24T01:15:37.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:37.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:37.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:37.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:37.134 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:37.135 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:37.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:37.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:37.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:37.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:37.136 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:37.457 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:37.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:37.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:37.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:37.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:37.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:37.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:37.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:37.903 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:37.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:37.904 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:37.904 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:37.905 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:38.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=13241 2020-08-24T01:15:38.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 13241 '!=' null 2020-08-24T01:15:38.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:38.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:38.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:38.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:38.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:38.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:38.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:38.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:38.496 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:38.496 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:38.496 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:38.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:38.845 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:38.845 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:38.845 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:39.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:39.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:39.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:39.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:39.289 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:39.289 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:39.289 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:39.290 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:39.290 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:39.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24598 2020-08-24T01:15:39.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24598 '!=' null 2020-08-24T01:15:39.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:39.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:39.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:39.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:39.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:39.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:39.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:39.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:39.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:39.923 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:39.923 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:40.272 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:40.273 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:40.273 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:40.273 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:40.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:40.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:40.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:40.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:40.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:40.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:40.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:40.725 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:40.725 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:41.216 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24598 2020-08-24T01:15:41.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24598 '!=' null 2020-08-24T01:15:41.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:41.217 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:41.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:41.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:41.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:41.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:41.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:41.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:41.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:41.320 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:41.320 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:41.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:41.666 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:41.667 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:41.667 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:42.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:42.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:42.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:42.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:42.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:42.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:42.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:42.114 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:42.114 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:42.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23397 2020-08-24T01:15:42.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23397 '!=' null 2020-08-24T01:15:42.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:42.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:42.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:42.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:42.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:42.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:42.679 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:42.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:42.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:42.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:42.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:43.014 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:43.015 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:43.015 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:43.016 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:43.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:43.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:43.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:43.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:43.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:43.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:43.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:43.488 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:43.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:43.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=36676 2020-08-24T01:15:43.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 36676 '!=' null 2020-08-24T01:15:43.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:43.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:43.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:44.068 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:44.069 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:44.070 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:44.070 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:44.070 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:44.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:44.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:44.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:44.419 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:44.419 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:44.420 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:44.420 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:44.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:44.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:44.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:44.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:44.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:44.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:44.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:44.865 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:44.866 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:45.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:15:45.295 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:15:45.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:45.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:45.296 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:45.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:45.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:45.398 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:45.398 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:45.399 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:45.399 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:45.399 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:45.410 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:45.729 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:45.731 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:45.731 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:45.731 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:46.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:46.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:46.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:46.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:46.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:46.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:46.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:46.167 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:46.167 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:46.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:15:46.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:15:46.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:46.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:46.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:46.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:46.711 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:46.712 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:46.713 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:46.713 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:46.713 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:46.714 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:46.714 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:47.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:47.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:47.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:47.054 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:47.496 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:47.497 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:47.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:47.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:47.498 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:47.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:47.499 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:47.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:15:47.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:15:47.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:47.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:47.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:48.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:48.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:48.035 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:48.035 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:48.035 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:48.035 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:48.035 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:48.036 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:48.381 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:48.382 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:48.382 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:48.383 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:48.812 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:48.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:48.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:48.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:48.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:48.813 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:48.814 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:48.814 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:48.814 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:49.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39206 2020-08-24T01:15:49.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39206 '!=' null 2020-08-24T01:15:49.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:49.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:49.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:49.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:49.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:49.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:49.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:49.371 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:49.371 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:49.371 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:49.372 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:49.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:49.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:49.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:49.720 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:50.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:50.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:50.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:50.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:50.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:50.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:50.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:50.161 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:50.161 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:50.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39206 2020-08-24T01:15:50.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39206 '!=' null 2020-08-24T01:15:50.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:50.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:50.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:50.726 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:50.726 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:50.727 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:50.728 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:50.728 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:50.728 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:50.728 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:50.729 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:51.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:51.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:51.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:51.094 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:51.538 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:51.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:51.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:51.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:51.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:51.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:51.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:51.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:51.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:51.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26559 2020-08-24T01:15:51.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26559 '!=' null 2020-08-24T01:15:51.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:51.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:51.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:52.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:52.079 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:52.080 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:52.080 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:52.080 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:52.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:52.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:52.084 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:52.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:52.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:52.440 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:52.441 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:52.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:52.867 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:52.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:52.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:52.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:52.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:52.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:52.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:52.870 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:53.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39207 2020-08-24T01:15:53.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39207 '!=' null 2020-08-24T01:15:53.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:53.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:53.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:53.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:53.412 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:53.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:53.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:53.412 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:53.413 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:53.413 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:53.413 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:53.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:53.748 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:53.749 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:53.749 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:54.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:54.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:54.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:54.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:54.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:54.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:54.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:54.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:54.190 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:54.640 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39207 2020-08-24T01:15:54.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39207 '!=' null 2020-08-24T01:15:54.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:54.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:54.641 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:54.747 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:54.747 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:54.747 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:54.747 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:54.748 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:54.748 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:54.748 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:54.748 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:55.099 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:55.100 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:55.100 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:55.101 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:55.545 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:55.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:55.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:55.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:55.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:55.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:55.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:55.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:55.548 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:56.015 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:15:56.015 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:15:56.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:56.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:56.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:56.122 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:56.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:56.123 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:56.124 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:56.124 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:56.124 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:56.125 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:56.125 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:56.469 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:56.471 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:56.472 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:56.472 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:56.972 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:56.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:56.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:56.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:56.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:56.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:56.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:56.977 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:56.977 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:57.437 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:15:57.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:15:57.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:57.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:57.438 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:57.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:57.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:57.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:57.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:57.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:57.540 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:57.541 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:57.541 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:57.893 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:57.894 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:57.894 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:57.895 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:58.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:58.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:58.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:58.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:58.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:58.364 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:58.364 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:58.364 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:58.364 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:15:58.816 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:15:58.817 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:15:58.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:15:58.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:15:58.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:15:58.916 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:15:58.917 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:15:58.918 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:15:58.918 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:15:58.918 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:15:58.919 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:15:58.919 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:15:58.919 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:15:59.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:15:59.284 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:15:59.284 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:15:59.284 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:15:59.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:15:59.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:15:59.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:15:59.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:15:59.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:15:59.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:15:59.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:15:59.719 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:15:59.720 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:00.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39206 2020-08-24T01:16:00.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39206 '!=' null 2020-08-24T01:16:00.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:00.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:00.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:00.291 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:00.291 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:00.292 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:00.292 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:00.293 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:00.293 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:00.293 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:00.293 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:00.648 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:16:00.649 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:00.650 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:00.650 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:01.145 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:16:01.146 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:16:01.146 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:16:01.146 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:16:01.146 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:16:01.147 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:16:01.147 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:16:01.147 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:16:01.148 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:01.592 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:16:01.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:16:01.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:01.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:01.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:01.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:01.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:01.695 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:01.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:01.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:01.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:01.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:01.696 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:02.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:16:02.027 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:02.027 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:02.027 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:02.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:16:02.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:16:02.472 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:16:02.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:16:02.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:16:02.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:16:02.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:16:02.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:16:02.474 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:02.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=25926 2020-08-24T01:16:02.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 25926 '!=' null 2020-08-24T01:16:02.955 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:02.956 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:02.956 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:03.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:03.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:03.058 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:03.058 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:03.058 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:03.059 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:03.059 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:03.059 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:03.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:16:03.396 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:03.397 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:03.397 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:03.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:16:03.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:16:03.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:16:03.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:16:03.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:16:03.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:16:03.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:16:03.847 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:16:03.847 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:04.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=39838 2020-08-24T01:16:04.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 39838 '!=' null 2020-08-24T01:16:04.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:04.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:04.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:04.400 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:04.401 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:04.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:04.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:04.402 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:04.403 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:04.403 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:04.403 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:04.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:16:04.738 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:04.738 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:04.738 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:05.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:16:05.186 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:16:05.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:16:05.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:16:05.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:16:05.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:16:05.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:16:05.189 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:16:05.189 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:05.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32629 2020-08-24T01:16:05.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32629 '!=' null 2020-08-24T01:16:05.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:05.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:05.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:05.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:05.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:05.791 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:05.791 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:05.792 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:05.792 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:05.792 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:05.792 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:06.149 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:16:06.149 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:06.150 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:06.150 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:06.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:16:06.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:16:06.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:16:06.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:16:06.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:16:06.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:16:06.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:16:06.608 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:16:06.609 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:07.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32629 2020-08-24T01:16:07.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32629 '!=' null 2020-08-24T01:16:07.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:07.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:07.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:07.127 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:07.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:07.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:07.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:07.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:07.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:07.129 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:07.130 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:07.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:16:07.477 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:07.478 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:07.478 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:07.936 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:16:07.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:16:07.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:16:07.937 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:16:07.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:16:07.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:16:07.938 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:16:07.938 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:16:07.939 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:08.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20867 2020-08-24T01:16:08.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20867 '!=' null 2020-08-24T01:16:08.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:08.387 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:08.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:08.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:08.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:08.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:08.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:08.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:08.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:08.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:08.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:08.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:16:08.854 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:08.854 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:08.854 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:09.310 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:16:09.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:16:09.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:16:09.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:16:09.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:16:09.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:16:09.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:16:09.312 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:16:09.313 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:16:09.746 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28454 2020-08-24T01:16:09.747 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28454 '!=' null 2020-08-24T01:16:09.747 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:16:09.747 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:16:09.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:16:09.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:16:09.849 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:16:09.850 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:16:09.850 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:16:09.850 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:16:09.850 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:16:09.851 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:16:09.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:16:10.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:16:10.203 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:16:10.203 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:16:10.203 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:16:10.649 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T01:16:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:16:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:16:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2020-08-24T01:16:10.650 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:16:10.650 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:16:11.008 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:16:11.008 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:16:11.008 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:16:11.008 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:16:11.009 INFO:tasks.workunit.client.0.smithi060.stderr:4 2020-08-24T01:16:11.009 INFO:tasks.workunit.client.0.smithi060.stderr:5' 2020-08-24T01:16:11.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:16:11.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:16:11.010 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:16:11.156 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836505 2020-08-24T01:16:11.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836505 2020-08-24T01:16:11.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505' 2020-08-24T01:16:11.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:16:11.158 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:16:11.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705688 2020-08-24T01:16:11.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 38654705688 2020-08-24T01:16:11.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-38654705688' 2020-08-24T01:16:11.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:16:11.304 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:16:11.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542168 2020-08-24T01:16:11.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 60129542168 2020-08-24T01:16:11.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-38654705688 2-60129542168' 2020-08-24T01:16:11.461 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:16:11.461 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:16:11.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378645 2020-08-24T01:16:11.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 81604378645 2020-08-24T01:16:11.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-38654705688 2-60129542168 3-81604378645' 2020-08-24T01:16:11.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:16:11.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:16:11.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215124 2020-08-24T01:16:11.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 103079215124 2020-08-24T01:16:11.762 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-38654705688 2-60129542168 3-81604378645 4-103079215124' 2020-08-24T01:16:11.763 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:16:11.763 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-08-24T01:16:11.920 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051602 2020-08-24T01:16:11.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 124554051602 2020-08-24T01:16:11.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836505 1-38654705688 2-60129542168 3-81604378645 4-103079215124 5-124554051602' 2020-08-24T01:16:11.921 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:16:11.921 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836505 2020-08-24T01:16:11.922 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:16:11.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:16:11.923 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:16:11.923 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836505 2020-08-24T01:16:11.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836505 2020-08-24T01:16:11.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836505' 2020-08-24T01:16:11.924 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:16:11.924 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836505 2020-08-24T01:16:12.286 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836503 -lt 21474836505 2020-08-24T01:16:12.286 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2084: flush_pg_stats: sleep 1 2020-08-24T01:16:13.288 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2085: flush_pg_stats: '[' 300 -eq 0 ']' 2020-08-24T01:16:13.288 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:16:13.654 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836506 -lt 21474836505 2020-08-24T01:16:13.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:16:13.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-38654705688 2020-08-24T01:16:13.656 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:16:13.657 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:16:13.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-38654705688 2020-08-24T01:16:13.658 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:16:13.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=38654705688 2020-08-24T01:16:13.660 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 38654705688' 2020-08-24T01:16:13.660 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 38654705688 2020-08-24T01:16:13.661 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:16:14.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 38654705688 -lt 38654705688 2020-08-24T01:16:14.041 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:16:14.042 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-60129542168 2020-08-24T01:16:14.042 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:16:14.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:16:14.044 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-60129542168 2020-08-24T01:16:14.044 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:16:14.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=60129542168 2020-08-24T01:16:14.046 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 60129542168' 2020-08-24T01:16:14.047 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 60129542168 2020-08-24T01:16:14.047 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:16:14.417 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 60129542168 -lt 60129542168 2020-08-24T01:16:14.418 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:16:14.418 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-81604378645 2020-08-24T01:16:14.419 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:16:14.421 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:16:14.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-81604378645 2020-08-24T01:16:14.422 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:16:14.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=81604378645 2020-08-24T01:16:14.423 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 81604378645' 2020-08-24T01:16:14.424 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 81604378645 2020-08-24T01:16:14.424 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:16:14.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 81604378646 -lt 81604378645 2020-08-24T01:16:14.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:16:14.789 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-103079215124 2020-08-24T01:16:14.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:16:14.791 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:16:14.792 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-103079215124 2020-08-24T01:16:14.793 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:16:14.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=103079215124 2020-08-24T01:16:14.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 103079215124' 2020-08-24T01:16:14.795 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 103079215124 2020-08-24T01:16:14.795 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:16:15.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 103079215124 -lt 103079215124 2020-08-24T01:16:15.168 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:16:15.168 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 5-124554051602 2020-08-24T01:16:15.169 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:16:15.171 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=5 2020-08-24T01:16:15.172 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-124554051602 2020-08-24T01:16:15.173 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:16:15.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=124554051602 2020-08-24T01:16:15.174 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.5 seq 124554051602' 2020-08-24T01:16:15.175 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.5 seq 124554051602 2020-08-24T01:16:15.175 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 5 2020-08-24T01:16:15.557 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 124554051603 -lt 124554051602 2020-08-24T01:16:15.557 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2020-08-24T01:16:15.558 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2020-08-24T01:16:15.922 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:16:15.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2020-08-24T01:16:15.932 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2020-08-24T01:16:15.933 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:16:15.933 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:16:15.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:16:15.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:16:15.934 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:16:16.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:16:16.253 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2020-08-24T01:16:16.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-08-24T01:16:16.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:16.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-08-24T01:16:16.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:16:16.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-08-24T01:16:16.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:16:16.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-08-24T01:16:16.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-08-24T01:16:16.898 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:16:16.898 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_log_t": { 2020-08-24T01:16:16.898 INFO:tasks.workunit.client.0.smithi060.stdout: "head": "33'150", 2020-08-24T01:16:16.899 INFO:tasks.workunit.client.0.smithi060.stdout: "tail": "33'148", 2020-08-24T01:16:16.899 INFO:tasks.workunit.client.0.smithi060.stdout: "log": [ 2020-08-24T01:16:16.899 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.899 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:16:16.899 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-08-24T01:16:16.900 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'149", 2020-08-24T01:16:16.900 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:16:16.900 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4635.0:1", 2020-08-24T01:16:16.900 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:16:16.900 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:15:25.634431+0000", 2020-08-24T01:16:16.901 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:16:16.901 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:16:16.901 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:16:16.901 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:16:16.901 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:16:16.902 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:16:16.902 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:16.902 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.902 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:16:16.902 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:16:16.903 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:16:16.903 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:16:16.903 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:16:16.903 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:16.903 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:16.903 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.904 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.904 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:16:16.904 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-08-24T01:16:16.904 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'150", 2020-08-24T01:16:16.904 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:16:16.905 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4638.0:1", 2020-08-24T01:16:16.905 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:16:16.905 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:15:25.773077+0000", 2020-08-24T01:16:16.905 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:16:16.905 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:16:16.906 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:16:16.906 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:16:16.906 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:16:16.906 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:16:16.906 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:16.907 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.907 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:16:16.907 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:16:16.907 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:16:16.907 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:16:16.908 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:16:16.908 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:16.908 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:16.908 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:16.908 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:16:16.909 INFO:tasks.workunit.client.0.smithi060.stdout: "dups": [ 2020-08-24T01:16:16.909 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.909 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4191.0:1", 2020-08-24T01:16:16.909 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'1", 2020-08-24T01:16:16.911 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "1", 2020-08-24T01:16:16.911 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.911 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.912 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.912 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4194.0:1", 2020-08-24T01:16:16.912 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'2", 2020-08-24T01:16:16.913 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "2", 2020-08-24T01:16:16.914 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.914 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.915 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.915 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4197.0:1", 2020-08-24T01:16:16.916 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'3", 2020-08-24T01:16:16.916 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "3", 2020-08-24T01:16:16.916 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.917 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.917 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.918 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4200.0:1", 2020-08-24T01:16:16.919 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'4", 2020-08-24T01:16:16.919 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "4", 2020-08-24T01:16:16.920 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.920 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.921 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.921 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4203.0:1", 2020-08-24T01:16:16.922 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'5", 2020-08-24T01:16:16.922 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "5", 2020-08-24T01:16:16.923 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.923 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.924 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.924 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4206.0:1", 2020-08-24T01:16:16.925 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'6", 2020-08-24T01:16:16.925 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "6", 2020-08-24T01:16:16.926 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.926 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.927 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.927 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4209.0:1", 2020-08-24T01:16:16.927 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'7", 2020-08-24T01:16:16.928 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "7", 2020-08-24T01:16:16.929 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.929 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.929 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.930 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4212.0:1", 2020-08-24T01:16:16.931 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'8", 2020-08-24T01:16:16.931 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "8", 2020-08-24T01:16:16.932 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.932 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.932 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.933 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4215.0:1", 2020-08-24T01:16:16.933 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'9", 2020-08-24T01:16:16.933 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "9", 2020-08-24T01:16:16.933 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.934 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.934 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.934 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4218.0:1", 2020-08-24T01:16:16.934 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'10", 2020-08-24T01:16:16.935 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "10", 2020-08-24T01:16:16.935 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.935 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.936 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.936 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4221.0:1", 2020-08-24T01:16:16.937 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'11", 2020-08-24T01:16:16.937 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "11", 2020-08-24T01:16:16.937 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.938 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.938 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.938 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4224.0:1", 2020-08-24T01:16:16.939 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'12", 2020-08-24T01:16:16.939 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "12", 2020-08-24T01:16:16.940 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.940 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.940 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.941 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4227.0:1", 2020-08-24T01:16:16.941 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'13", 2020-08-24T01:16:16.942 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "13", 2020-08-24T01:16:16.942 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.942 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.943 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.943 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4230.0:1", 2020-08-24T01:16:16.943 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'14", 2020-08-24T01:16:16.943 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "14", 2020-08-24T01:16:16.944 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.944 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.944 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.944 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4233.0:1", 2020-08-24T01:16:16.944 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'15", 2020-08-24T01:16:16.950 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "15", 2020-08-24T01:16:16.950 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.950 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.950 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.951 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4236.0:1", 2020-08-24T01:16:16.951 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'16", 2020-08-24T01:16:16.951 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "16", 2020-08-24T01:16:16.951 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.952 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.952 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.952 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4239.0:1", 2020-08-24T01:16:16.952 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'17", 2020-08-24T01:16:16.952 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "17", 2020-08-24T01:16:16.953 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.953 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.953 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.953 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4242.0:1", 2020-08-24T01:16:16.954 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'18", 2020-08-24T01:16:16.954 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "18", 2020-08-24T01:16:16.954 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.954 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.954 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4245.0:1", 2020-08-24T01:16:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'19", 2020-08-24T01:16:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "19", 2020-08-24T01:16:16.955 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4248.0:1", 2020-08-24T01:16:16.956 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'20", 2020-08-24T01:16:16.957 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "20", 2020-08-24T01:16:16.957 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.957 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.957 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.957 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4251.0:1", 2020-08-24T01:16:16.958 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'21", 2020-08-24T01:16:16.958 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "21", 2020-08-24T01:16:16.958 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.959 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.959 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.959 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4254.0:1", 2020-08-24T01:16:16.959 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'22", 2020-08-24T01:16:16.959 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "22", 2020-08-24T01:16:16.960 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.960 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.960 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.966 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4257.0:1", 2020-08-24T01:16:16.966 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'23", 2020-08-24T01:16:16.967 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "23", 2020-08-24T01:16:16.967 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.967 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.967 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.968 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4260.0:1", 2020-08-24T01:16:16.968 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'24", 2020-08-24T01:16:16.968 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "24", 2020-08-24T01:16:16.968 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.968 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.969 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.969 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4263.0:1", 2020-08-24T01:16:16.969 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'25", 2020-08-24T01:16:16.969 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "25", 2020-08-24T01:16:16.969 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.970 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.970 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.970 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4266.0:1", 2020-08-24T01:16:16.970 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'26", 2020-08-24T01:16:16.971 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "26", 2020-08-24T01:16:16.971 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.971 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.971 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.971 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4269.0:1", 2020-08-24T01:16:16.972 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'27", 2020-08-24T01:16:16.972 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "27", 2020-08-24T01:16:16.972 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.972 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.973 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.973 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4272.0:1", 2020-08-24T01:16:16.973 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'28", 2020-08-24T01:16:16.973 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "28", 2020-08-24T01:16:16.973 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.974 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.974 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.974 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4275.0:1", 2020-08-24T01:16:16.974 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'29", 2020-08-24T01:16:16.974 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "29", 2020-08-24T01:16:16.975 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.975 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.975 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.975 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4278.0:1", 2020-08-24T01:16:16.976 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'30", 2020-08-24T01:16:16.976 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "30", 2020-08-24T01:16:16.976 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.976 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.976 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.977 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4281.0:1", 2020-08-24T01:16:16.977 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'31", 2020-08-24T01:16:16.977 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "31", 2020-08-24T01:16:16.977 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.978 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.978 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.978 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4284.0:1", 2020-08-24T01:16:16.978 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'32", 2020-08-24T01:16:16.979 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "32", 2020-08-24T01:16:16.979 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.979 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.979 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.979 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4287.0:1", 2020-08-24T01:16:16.980 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'33", 2020-08-24T01:16:16.980 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "33", 2020-08-24T01:16:16.980 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.980 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.981 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.981 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4290.0:1", 2020-08-24T01:16:16.981 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'34", 2020-08-24T01:16:16.981 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "34", 2020-08-24T01:16:16.988 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.988 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.989 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.989 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4293.0:1", 2020-08-24T01:16:16.989 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'35", 2020-08-24T01:16:16.989 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "35", 2020-08-24T01:16:16.990 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.990 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.990 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.990 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4296.0:1", 2020-08-24T01:16:16.990 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'36", 2020-08-24T01:16:16.990 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "36", 2020-08-24T01:16:16.991 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.991 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.991 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.991 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4299.0:1", 2020-08-24T01:16:16.991 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'37", 2020-08-24T01:16:16.992 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "37", 2020-08-24T01:16:16.992 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.992 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.992 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.992 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4302.0:1", 2020-08-24T01:16:16.993 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'38", 2020-08-24T01:16:16.993 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "38", 2020-08-24T01:16:16.995 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.995 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.996 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.996 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4305.0:1", 2020-08-24T01:16:16.996 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'39", 2020-08-24T01:16:16.996 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "39", 2020-08-24T01:16:16.996 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.996 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.997 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.997 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4308.0:1", 2020-08-24T01:16:16.997 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'40", 2020-08-24T01:16:16.997 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "40", 2020-08-24T01:16:16.998 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:16.998 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:16.998 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:16.999 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4311.0:1", 2020-08-24T01:16:16.999 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'41", 2020-08-24T01:16:17.000 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "41", 2020-08-24T01:16:17.000 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.000 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.000 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.001 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4314.0:1", 2020-08-24T01:16:17.001 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'42", 2020-08-24T01:16:17.001 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "42", 2020-08-24T01:16:17.001 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.001 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.002 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.002 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4317.0:1", 2020-08-24T01:16:17.002 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'43", 2020-08-24T01:16:17.002 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "43", 2020-08-24T01:16:17.002 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.003 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.003 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.003 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4320.0:1", 2020-08-24T01:16:17.003 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'44", 2020-08-24T01:16:17.003 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "44", 2020-08-24T01:16:17.003 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.004 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.004 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.004 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4323.0:1", 2020-08-24T01:16:17.004 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'45", 2020-08-24T01:16:17.004 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "45", 2020-08-24T01:16:17.005 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.005 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.005 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.005 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4326.0:1", 2020-08-24T01:16:17.005 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'46", 2020-08-24T01:16:17.006 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "46", 2020-08-24T01:16:17.006 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.006 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.007 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.007 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4329.0:1", 2020-08-24T01:16:17.008 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'47", 2020-08-24T01:16:17.008 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "47", 2020-08-24T01:16:17.008 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.008 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.008 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.009 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4332.0:1", 2020-08-24T01:16:17.009 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'48", 2020-08-24T01:16:17.009 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "48", 2020-08-24T01:16:17.009 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.009 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.010 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.010 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4335.0:1", 2020-08-24T01:16:17.010 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'49", 2020-08-24T01:16:17.010 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "49", 2020-08-24T01:16:17.010 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.011 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.011 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.011 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4338.0:1", 2020-08-24T01:16:17.011 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'50", 2020-08-24T01:16:17.011 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "50", 2020-08-24T01:16:17.011 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.012 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.012 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.012 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4341.0:1", 2020-08-24T01:16:17.012 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'51", 2020-08-24T01:16:17.012 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "51", 2020-08-24T01:16:17.013 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.013 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.013 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.013 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4344.0:1", 2020-08-24T01:16:17.013 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'52", 2020-08-24T01:16:17.014 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "52", 2020-08-24T01:16:17.014 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.014 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.014 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.014 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4347.0:1", 2020-08-24T01:16:17.015 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'53", 2020-08-24T01:16:17.015 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "53", 2020-08-24T01:16:17.015 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.015 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.015 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.016 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4350.0:1", 2020-08-24T01:16:17.016 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'54", 2020-08-24T01:16:17.016 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "54", 2020-08-24T01:16:17.016 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.017 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.017 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.017 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4353.0:1", 2020-08-24T01:16:17.017 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'55", 2020-08-24T01:16:17.017 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "55", 2020-08-24T01:16:17.018 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.018 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.018 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.018 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4356.0:1", 2020-08-24T01:16:17.018 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'56", 2020-08-24T01:16:17.018 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "56", 2020-08-24T01:16:17.019 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.019 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.019 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.019 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4359.0:1", 2020-08-24T01:16:17.019 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'57", 2020-08-24T01:16:17.020 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "57", 2020-08-24T01:16:17.020 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.020 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.020 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.020 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4362.0:1", 2020-08-24T01:16:17.020 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'58", 2020-08-24T01:16:17.021 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "58", 2020-08-24T01:16:17.021 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.021 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.021 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.021 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4365.0:1", 2020-08-24T01:16:17.022 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'59", 2020-08-24T01:16:17.022 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "59", 2020-08-24T01:16:17.022 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.022 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.022 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.022 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4368.0:1", 2020-08-24T01:16:17.023 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'60", 2020-08-24T01:16:17.023 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "60", 2020-08-24T01:16:17.023 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.023 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.023 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.024 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4371.0:1", 2020-08-24T01:16:17.024 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'61", 2020-08-24T01:16:17.024 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "61", 2020-08-24T01:16:17.024 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.024 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.025 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.025 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4374.0:1", 2020-08-24T01:16:17.025 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'62", 2020-08-24T01:16:17.025 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "62", 2020-08-24T01:16:17.025 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.025 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.026 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.026 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4377.0:1", 2020-08-24T01:16:17.026 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'63", 2020-08-24T01:16:17.026 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "63", 2020-08-24T01:16:17.026 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.027 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.027 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.027 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4380.0:1", 2020-08-24T01:16:17.027 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'64", 2020-08-24T01:16:17.027 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "64", 2020-08-24T01:16:17.027 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.028 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.028 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.028 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4383.0:1", 2020-08-24T01:16:17.028 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'65", 2020-08-24T01:16:17.028 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "65", 2020-08-24T01:16:17.029 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.029 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.029 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.029 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4386.0:1", 2020-08-24T01:16:17.029 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'66", 2020-08-24T01:16:17.030 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "66", 2020-08-24T01:16:17.030 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.030 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.030 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.030 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4389.0:1", 2020-08-24T01:16:17.030 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'67", 2020-08-24T01:16:17.031 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "67", 2020-08-24T01:16:17.031 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.031 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.031 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.031 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4392.0:1", 2020-08-24T01:16:17.032 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'68", 2020-08-24T01:16:17.032 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "68", 2020-08-24T01:16:17.032 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.032 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.032 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.032 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4395.0:1", 2020-08-24T01:16:17.033 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'69", 2020-08-24T01:16:17.033 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "69", 2020-08-24T01:16:17.033 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.033 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.033 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.034 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4398.0:1", 2020-08-24T01:16:17.034 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'70", 2020-08-24T01:16:17.034 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "70", 2020-08-24T01:16:17.034 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.034 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.034 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.035 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4401.0:1", 2020-08-24T01:16:17.035 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'71", 2020-08-24T01:16:17.035 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "71", 2020-08-24T01:16:17.035 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.035 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.036 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.036 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4404.0:1", 2020-08-24T01:16:17.036 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'72", 2020-08-24T01:16:17.036 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "72", 2020-08-24T01:16:17.036 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.036 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.037 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.037 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4407.0:1", 2020-08-24T01:16:17.037 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'73", 2020-08-24T01:16:17.037 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "73", 2020-08-24T01:16:17.037 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.038 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.038 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.038 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4410.0:1", 2020-08-24T01:16:17.038 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'74", 2020-08-24T01:16:17.039 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "74", 2020-08-24T01:16:17.039 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.039 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.039 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.039 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4413.0:1", 2020-08-24T01:16:17.040 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'75", 2020-08-24T01:16:17.040 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "75", 2020-08-24T01:16:17.040 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.040 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.040 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.040 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4416.0:1", 2020-08-24T01:16:17.041 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'76", 2020-08-24T01:16:17.041 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "76", 2020-08-24T01:16:17.041 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.041 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.041 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.042 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4419.0:1", 2020-08-24T01:16:17.042 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'77", 2020-08-24T01:16:17.042 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "77", 2020-08-24T01:16:17.042 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.042 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.042 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.043 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4422.0:1", 2020-08-24T01:16:17.043 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'78", 2020-08-24T01:16:17.043 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "78", 2020-08-24T01:16:17.043 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.044 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.044 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.044 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4425.0:1", 2020-08-24T01:16:17.044 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'79", 2020-08-24T01:16:17.044 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "79", 2020-08-24T01:16:17.045 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.045 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.045 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.045 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4428.0:1", 2020-08-24T01:16:17.045 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'80", 2020-08-24T01:16:17.046 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "80", 2020-08-24T01:16:17.046 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.046 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.046 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.046 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4431.0:1", 2020-08-24T01:16:17.047 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'81", 2020-08-24T01:16:17.047 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "81", 2020-08-24T01:16:17.047 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.047 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.047 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.048 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4434.0:1", 2020-08-24T01:16:17.048 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'82", 2020-08-24T01:16:17.048 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "82", 2020-08-24T01:16:17.048 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.048 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.049 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.049 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4437.0:1", 2020-08-24T01:16:17.049 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'83", 2020-08-24T01:16:17.049 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "83", 2020-08-24T01:16:17.049 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.050 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.050 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.050 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4440.0:1", 2020-08-24T01:16:17.050 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'84", 2020-08-24T01:16:17.050 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "84", 2020-08-24T01:16:17.050 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.051 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.051 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.051 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4443.0:1", 2020-08-24T01:16:17.051 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'85", 2020-08-24T01:16:17.051 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "85", 2020-08-24T01:16:17.052 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.052 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.052 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.052 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4446.0:1", 2020-08-24T01:16:17.052 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'86", 2020-08-24T01:16:17.052 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "86", 2020-08-24T01:16:17.053 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.053 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.053 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.053 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4449.0:1", 2020-08-24T01:16:17.053 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'87", 2020-08-24T01:16:17.054 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "87", 2020-08-24T01:16:17.054 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.054 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.054 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.054 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4452.0:1", 2020-08-24T01:16:17.054 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'88", 2020-08-24T01:16:17.055 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "88", 2020-08-24T01:16:17.055 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.055 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.055 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.055 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4455.0:1", 2020-08-24T01:16:17.056 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'89", 2020-08-24T01:16:17.056 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "89", 2020-08-24T01:16:17.056 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.056 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.056 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.056 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4458.0:1", 2020-08-24T01:16:17.057 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'90", 2020-08-24T01:16:17.057 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "90", 2020-08-24T01:16:17.057 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.057 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.057 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.058 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4461.0:1", 2020-08-24T01:16:17.058 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'91", 2020-08-24T01:16:17.058 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "91", 2020-08-24T01:16:17.058 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.058 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.059 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.059 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4464.0:1", 2020-08-24T01:16:17.059 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'92", 2020-08-24T01:16:17.059 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "92", 2020-08-24T01:16:17.059 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.059 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.060 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.060 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4467.0:1", 2020-08-24T01:16:17.060 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'93", 2020-08-24T01:16:17.060 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "93", 2020-08-24T01:16:17.060 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.060 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.061 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.061 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4470.0:1", 2020-08-24T01:16:17.061 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'94", 2020-08-24T01:16:17.061 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "94", 2020-08-24T01:16:17.061 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.062 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.062 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.062 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4473.0:1", 2020-08-24T01:16:17.063 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'95", 2020-08-24T01:16:17.063 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "95", 2020-08-24T01:16:17.063 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.063 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.063 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.063 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4476.0:1", 2020-08-24T01:16:17.064 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'96", 2020-08-24T01:16:17.064 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "96", 2020-08-24T01:16:17.064 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.064 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.064 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.065 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4479.0:1", 2020-08-24T01:16:17.065 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'97", 2020-08-24T01:16:17.065 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "97", 2020-08-24T01:16:17.065 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.065 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.065 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.066 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4482.0:1", 2020-08-24T01:16:17.066 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'98", 2020-08-24T01:16:17.066 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "98", 2020-08-24T01:16:17.066 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.066 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.067 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.067 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4485.0:1", 2020-08-24T01:16:17.067 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'99", 2020-08-24T01:16:17.067 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "99", 2020-08-24T01:16:17.067 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.067 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.068 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.068 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4488.0:1", 2020-08-24T01:16:17.068 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'100", 2020-08-24T01:16:17.068 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "100", 2020-08-24T01:16:17.068 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.069 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.069 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.069 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4491.0:1", 2020-08-24T01:16:17.069 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'101", 2020-08-24T01:16:17.069 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "101", 2020-08-24T01:16:17.070 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.070 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.070 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.070 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4494.0:1", 2020-08-24T01:16:17.070 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'102", 2020-08-24T01:16:17.070 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "102", 2020-08-24T01:16:17.071 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.071 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.071 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.071 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4497.0:1", 2020-08-24T01:16:17.071 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'103", 2020-08-24T01:16:17.072 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "103", 2020-08-24T01:16:17.072 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.072 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.072 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.072 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4500.0:1", 2020-08-24T01:16:17.073 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'104", 2020-08-24T01:16:17.073 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "104", 2020-08-24T01:16:17.073 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.073 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.073 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.074 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4503.0:1", 2020-08-24T01:16:17.074 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'105", 2020-08-24T01:16:17.074 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "105", 2020-08-24T01:16:17.074 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.074 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.075 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.075 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4506.0:1", 2020-08-24T01:16:17.075 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'106", 2020-08-24T01:16:17.075 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "106", 2020-08-24T01:16:17.075 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.075 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.076 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.076 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4509.0:1", 2020-08-24T01:16:17.076 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'107", 2020-08-24T01:16:17.076 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "107", 2020-08-24T01:16:17.076 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.077 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.077 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.077 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4512.0:1", 2020-08-24T01:16:17.077 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'108", 2020-08-24T01:16:17.077 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "108", 2020-08-24T01:16:17.077 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.078 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.078 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.078 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4515.0:1", 2020-08-24T01:16:17.078 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'109", 2020-08-24T01:16:17.078 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "109", 2020-08-24T01:16:17.079 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.079 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.079 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.079 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4518.0:1", 2020-08-24T01:16:17.079 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'110", 2020-08-24T01:16:17.079 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "110", 2020-08-24T01:16:17.080 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.080 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.080 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.080 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4521.0:1", 2020-08-24T01:16:17.080 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'111", 2020-08-24T01:16:17.081 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "111", 2020-08-24T01:16:17.081 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.081 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.081 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.081 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4524.0:1", 2020-08-24T01:16:17.081 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'112", 2020-08-24T01:16:17.082 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "112", 2020-08-24T01:16:17.082 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.082 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.082 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.082 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4527.0:1", 2020-08-24T01:16:17.083 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'113", 2020-08-24T01:16:17.083 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "113", 2020-08-24T01:16:17.083 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.083 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.083 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.083 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4530.0:1", 2020-08-24T01:16:17.084 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'114", 2020-08-24T01:16:17.084 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "114", 2020-08-24T01:16:17.084 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.084 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.084 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.085 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4533.0:1", 2020-08-24T01:16:17.085 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'115", 2020-08-24T01:16:17.085 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "115", 2020-08-24T01:16:17.085 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.085 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.085 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.086 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4536.0:1", 2020-08-24T01:16:17.086 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'116", 2020-08-24T01:16:17.086 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "116", 2020-08-24T01:16:17.086 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.086 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.087 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.087 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4539.0:1", 2020-08-24T01:16:17.087 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'117", 2020-08-24T01:16:17.087 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "117", 2020-08-24T01:16:17.087 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.087 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.088 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.088 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4542.0:1", 2020-08-24T01:16:17.088 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'118", 2020-08-24T01:16:17.088 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "118", 2020-08-24T01:16:17.089 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.089 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.089 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.089 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4545.0:1", 2020-08-24T01:16:17.089 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'119", 2020-08-24T01:16:17.090 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "119", 2020-08-24T01:16:17.090 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.090 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.090 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.090 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4548.0:1", 2020-08-24T01:16:17.090 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'120", 2020-08-24T01:16:17.091 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "120", 2020-08-24T01:16:17.091 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.091 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.091 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.092 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4551.0:1", 2020-08-24T01:16:17.092 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'121", 2020-08-24T01:16:17.092 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "121", 2020-08-24T01:16:17.092 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.092 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.093 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.093 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4554.0:1", 2020-08-24T01:16:17.093 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'122", 2020-08-24T01:16:17.093 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "122", 2020-08-24T01:16:17.093 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.093 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.094 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.094 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4557.0:1", 2020-08-24T01:16:17.094 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'123", 2020-08-24T01:16:17.094 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "123", 2020-08-24T01:16:17.094 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.095 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.095 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.095 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4560.0:1", 2020-08-24T01:16:17.095 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'124", 2020-08-24T01:16:17.095 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "124", 2020-08-24T01:16:17.095 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.096 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.096 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.096 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4563.0:1", 2020-08-24T01:16:17.096 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'125", 2020-08-24T01:16:17.096 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "125", 2020-08-24T01:16:17.097 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.097 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.097 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.097 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4566.0:1", 2020-08-24T01:16:17.097 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'126", 2020-08-24T01:16:17.098 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "126", 2020-08-24T01:16:17.098 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.098 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.098 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.098 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4569.0:1", 2020-08-24T01:16:17.098 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'127", 2020-08-24T01:16:17.099 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "127", 2020-08-24T01:16:17.099 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.099 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.099 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.099 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4572.0:1", 2020-08-24T01:16:17.099 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'128", 2020-08-24T01:16:17.100 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "128", 2020-08-24T01:16:17.100 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.100 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.100 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.100 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4575.0:1", 2020-08-24T01:16:17.101 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'129", 2020-08-24T01:16:17.101 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "129", 2020-08-24T01:16:17.101 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.101 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.101 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.101 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4578.0:1", 2020-08-24T01:16:17.102 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'130", 2020-08-24T01:16:17.102 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "130", 2020-08-24T01:16:17.102 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.102 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.102 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.103 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4581.0:1", 2020-08-24T01:16:17.103 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'131", 2020-08-24T01:16:17.103 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "131", 2020-08-24T01:16:17.103 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.103 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.104 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.104 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4584.0:1", 2020-08-24T01:16:17.104 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'132", 2020-08-24T01:16:17.104 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "132", 2020-08-24T01:16:17.104 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.104 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.105 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.105 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4587.0:1", 2020-08-24T01:16:17.105 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'133", 2020-08-24T01:16:17.105 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "133", 2020-08-24T01:16:17.105 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.106 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.106 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.106 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4590.0:1", 2020-08-24T01:16:17.106 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'134", 2020-08-24T01:16:17.106 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "134", 2020-08-24T01:16:17.106 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.107 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.107 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.107 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4593.0:1", 2020-08-24T01:16:17.107 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'135", 2020-08-24T01:16:17.107 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "135", 2020-08-24T01:16:17.108 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.108 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.108 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.108 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4596.0:1", 2020-08-24T01:16:17.108 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'136", 2020-08-24T01:16:17.108 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "136", 2020-08-24T01:16:17.109 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.109 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.109 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.109 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4599.0:1", 2020-08-24T01:16:17.109 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'137", 2020-08-24T01:16:17.110 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "137", 2020-08-24T01:16:17.110 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.110 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.110 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.110 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4602.0:1", 2020-08-24T01:16:17.110 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'138", 2020-08-24T01:16:17.111 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "138", 2020-08-24T01:16:17.111 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.111 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.111 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.111 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4605.0:1", 2020-08-24T01:16:17.112 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'139", 2020-08-24T01:16:17.112 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "139", 2020-08-24T01:16:17.112 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.112 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.112 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.112 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4608.0:1", 2020-08-24T01:16:17.113 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'140", 2020-08-24T01:16:17.113 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "140", 2020-08-24T01:16:17.113 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.113 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.113 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.114 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4611.0:1", 2020-08-24T01:16:17.114 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'141", 2020-08-24T01:16:17.114 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "141", 2020-08-24T01:16:17.114 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.114 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.114 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.115 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4614.0:1", 2020-08-24T01:16:17.115 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'142", 2020-08-24T01:16:17.115 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "142", 2020-08-24T01:16:17.115 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.115 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.116 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.116 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4617.0:1", 2020-08-24T01:16:17.116 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'143", 2020-08-24T01:16:17.116 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "143", 2020-08-24T01:16:17.116 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.117 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.117 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.117 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4620.0:1", 2020-08-24T01:16:17.117 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'144", 2020-08-24T01:16:17.118 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "144", 2020-08-24T01:16:17.118 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.118 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.118 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.118 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4623.0:1", 2020-08-24T01:16:17.118 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'145", 2020-08-24T01:16:17.119 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "145", 2020-08-24T01:16:17.119 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.119 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.119 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.119 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4626.0:1", 2020-08-24T01:16:17.120 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'146", 2020-08-24T01:16:17.120 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "146", 2020-08-24T01:16:17.120 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.120 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.120 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.120 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4629.0:1", 2020-08-24T01:16:17.121 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'147", 2020-08-24T01:16:17.434 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "147", 2020-08-24T01:16:17.435 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.435 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.436 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:16:17.436 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4632.0:1", 2020-08-24T01:16:17.436 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "33'148", 2020-08-24T01:16:17.437 INFO:tasks.workunit.client.0.smithi060.stdout: "user_version": "148", 2020-08-24T01:16:17.437 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": "0" 2020-08-24T01:16:17.437 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:17.437 INFO:tasks.workunit.client.0.smithi060.stdout: ] 2020-08-24T01:16:17.438 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:16:17.438 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_missing_t": { 2020-08-24T01:16:17.438 INFO:tasks.workunit.client.0.smithi060.stdout: "missing": [], 2020-08-24T01:16:17.439 INFO:tasks.workunit.client.0.smithi060.stdout: "may_include_deletes": true 2020-08-24T01:16:17.439 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:16:17.440 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:16:17.441 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-08-24T01:16:17.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=2 2020-08-24T01:16:17.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 2 '!=' 2 ']' 2020-08-24T01:16:17.447 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-08-24T01:16:17.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=148 2020-08-24T01:16:17.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 148 '!=' 148 ']' 2020-08-24T01:16:17.452 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-08-24T01:16:17.458 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to max 2 2020-08-24T01:16:17.458 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to copy log version 33'150 2020-08-24T01:16:17.458 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to copy log version 33'149 2020-08-24T01:16:17.459 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after earliest_dup_version 0 2020-08-24T01:16:17.459 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'1 2020-08-24T01:16:17.460 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'2 2020-08-24T01:16:17.460 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'3 2020-08-24T01:16:17.460 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'4 2020-08-24T01:16:17.461 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'5 2020-08-24T01:16:17.461 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'6 2020-08-24T01:16:17.461 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'7 2020-08-24T01:16:17.462 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'8 2020-08-24T01:16:17.462 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'9 2020-08-24T01:16:17.463 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'10 2020-08-24T01:16:17.463 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'11 2020-08-24T01:16:17.463 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'12 2020-08-24T01:16:17.464 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'13 2020-08-24T01:16:17.464 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'14 2020-08-24T01:16:17.464 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'15 2020-08-24T01:16:17.465 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'16 2020-08-24T01:16:17.465 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'17 2020-08-24T01:16:17.466 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'18 2020-08-24T01:16:17.466 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'19 2020-08-24T01:16:17.466 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'20 2020-08-24T01:16:17.467 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'21 2020-08-24T01:16:17.467 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'22 2020-08-24T01:16:17.467 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'23 2020-08-24T01:16:17.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'24 2020-08-24T01:16:17.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'25 2020-08-24T01:16:17.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'26 2020-08-24T01:16:17.468 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'27 2020-08-24T01:16:17.469 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'28 2020-08-24T01:16:17.469 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'29 2020-08-24T01:16:17.469 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'30 2020-08-24T01:16:17.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'31 2020-08-24T01:16:17.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'32 2020-08-24T01:16:17.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'33 2020-08-24T01:16:17.470 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'34 2020-08-24T01:16:17.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'35 2020-08-24T01:16:17.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'36 2020-08-24T01:16:17.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'37 2020-08-24T01:16:17.471 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'38 2020-08-24T01:16:17.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'39 2020-08-24T01:16:17.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'40 2020-08-24T01:16:17.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'41 2020-08-24T01:16:17.472 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'42 2020-08-24T01:16:17.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'43 2020-08-24T01:16:17.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'44 2020-08-24T01:16:17.473 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'45 2020-08-24T01:16:17.474 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'46 2020-08-24T01:16:17.474 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'47 2020-08-24T01:16:17.474 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'48 2020-08-24T01:16:17.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'49 2020-08-24T01:16:17.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'50 2020-08-24T01:16:17.475 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'51 2020-08-24T01:16:17.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'52 2020-08-24T01:16:17.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'53 2020-08-24T01:16:17.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'54 2020-08-24T01:16:17.476 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'55 2020-08-24T01:16:17.477 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'56 2020-08-24T01:16:17.477 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'57 2020-08-24T01:16:17.477 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'58 2020-08-24T01:16:17.477 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'59 2020-08-24T01:16:17.478 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'60 2020-08-24T01:16:17.478 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'61 2020-08-24T01:16:17.478 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'62 2020-08-24T01:16:17.479 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'63 2020-08-24T01:16:17.479 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'64 2020-08-24T01:16:17.479 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'65 2020-08-24T01:16:17.479 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'66 2020-08-24T01:16:17.479 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'67 2020-08-24T01:16:17.480 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'68 2020-08-24T01:16:17.480 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'69 2020-08-24T01:16:17.480 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'70 2020-08-24T01:16:17.480 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'71 2020-08-24T01:16:17.480 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'72 2020-08-24T01:16:17.480 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'73 2020-08-24T01:16:17.481 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'74 2020-08-24T01:16:17.481 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'75 2020-08-24T01:16:17.481 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'76 2020-08-24T01:16:17.481 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'77 2020-08-24T01:16:17.481 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'78 2020-08-24T01:16:17.482 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'79 2020-08-24T01:16:17.482 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'80 2020-08-24T01:16:17.482 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'81 2020-08-24T01:16:17.482 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'82 2020-08-24T01:16:17.483 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'83 2020-08-24T01:16:17.483 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'84 2020-08-24T01:16:17.483 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'85 2020-08-24T01:16:17.483 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'86 2020-08-24T01:16:17.483 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'87 2020-08-24T01:16:17.484 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'88 2020-08-24T01:16:17.484 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'89 2020-08-24T01:16:17.484 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'90 2020-08-24T01:16:17.484 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'91 2020-08-24T01:16:17.484 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'92 2020-08-24T01:16:17.485 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'93 2020-08-24T01:16:17.485 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'94 2020-08-24T01:16:17.485 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'95 2020-08-24T01:16:17.486 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'96 2020-08-24T01:16:17.487 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'97 2020-08-24T01:16:17.487 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'98 2020-08-24T01:16:17.487 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'99 2020-08-24T01:16:17.487 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'100 2020-08-24T01:16:17.487 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-08-24T01:16:17.488 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-08-24T01:16:17.488 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-08-24T01:16:17.488 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-08-24T01:16:17.489 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-08-24T01:16:17.489 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-08-24T01:16:17.489 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-08-24T01:16:17.489 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-08-24T01:16:17.489 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-08-24T01:16:17.490 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-08-24T01:16:17.490 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-08-24T01:16:17.490 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-08-24T01:16:17.490 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-08-24T01:16:17.490 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-08-24T01:16:17.491 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-08-24T01:16:17.491 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-08-24T01:16:17.491 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-08-24T01:16:17.491 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-08-24T01:16:17.491 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-08-24T01:16:17.492 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-08-24T01:16:17.492 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-08-24T01:16:17.492 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-08-24T01:16:17.492 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-08-24T01:16:17.492 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-08-24T01:16:17.493 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-08-24T01:16:17.493 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-08-24T01:16:17.493 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-08-24T01:16:17.493 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-08-24T01:16:17.494 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-08-24T01:16:17.494 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-08-24T01:16:17.494 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-08-24T01:16:17.494 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-08-24T01:16:17.495 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-08-24T01:16:17.495 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-08-24T01:16:17.495 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-08-24T01:16:17.495 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-08-24T01:16:17.495 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-08-24T01:16:17.496 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-08-24T01:16:17.496 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-08-24T01:16:17.496 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-08-24T01:16:17.496 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-08-24T01:16:17.496 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-08-24T01:16:17.497 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-08-24T01:16:17.497 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-08-24T01:16:17.497 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-08-24T01:16:17.497 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-08-24T01:16:17.497 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-08-24T01:16:17.498 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-08-24T01:16:17.498 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to max 2 2020-08-24T01:16:17.498 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to copy log version 33'150 2020-08-24T01:16:17.498 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to copy log version 33'149 2020-08-24T01:16:17.499 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after earliest_dup_version 0 2020-08-24T01:16:17.499 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'1 2020-08-24T01:16:17.499 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'2 2020-08-24T01:16:17.499 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'3 2020-08-24T01:16:17.499 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'4 2020-08-24T01:16:17.499 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'5 2020-08-24T01:16:17.500 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'6 2020-08-24T01:16:17.500 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'7 2020-08-24T01:16:17.500 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'8 2020-08-24T01:16:17.500 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'9 2020-08-24T01:16:17.500 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'10 2020-08-24T01:16:17.501 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'11 2020-08-24T01:16:17.501 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'12 2020-08-24T01:16:17.501 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'13 2020-08-24T01:16:17.501 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'14 2020-08-24T01:16:17.501 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'15 2020-08-24T01:16:17.502 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'16 2020-08-24T01:16:17.502 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'17 2020-08-24T01:16:17.502 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'18 2020-08-24T01:16:17.502 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'19 2020-08-24T01:16:17.502 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'20 2020-08-24T01:16:17.503 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'21 2020-08-24T01:16:17.503 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'22 2020-08-24T01:16:17.503 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'23 2020-08-24T01:16:17.503 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'24 2020-08-24T01:16:17.503 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'25 2020-08-24T01:16:17.504 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'26 2020-08-24T01:16:17.504 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'27 2020-08-24T01:16:17.504 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'28 2020-08-24T01:16:17.504 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'29 2020-08-24T01:16:17.504 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'30 2020-08-24T01:16:17.504 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'31 2020-08-24T01:16:17.505 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'32 2020-08-24T01:16:17.505 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'33 2020-08-24T01:16:17.505 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'34 2020-08-24T01:16:17.505 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'35 2020-08-24T01:16:17.505 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'36 2020-08-24T01:16:17.506 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'37 2020-08-24T01:16:17.506 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'38 2020-08-24T01:16:17.506 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'39 2020-08-24T01:16:17.506 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'40 2020-08-24T01:16:17.506 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'41 2020-08-24T01:16:17.507 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'42 2020-08-24T01:16:17.507 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'43 2020-08-24T01:16:17.507 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'44 2020-08-24T01:16:17.507 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'45 2020-08-24T01:16:17.507 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'46 2020-08-24T01:16:17.508 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'47 2020-08-24T01:16:17.508 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'48 2020-08-24T01:16:17.508 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'49 2020-08-24T01:16:17.508 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'50 2020-08-24T01:16:17.508 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'51 2020-08-24T01:16:17.509 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'52 2020-08-24T01:16:17.509 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'53 2020-08-24T01:16:17.509 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'54 2020-08-24T01:16:17.509 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'55 2020-08-24T01:16:17.509 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'56 2020-08-24T01:16:17.510 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'57 2020-08-24T01:16:17.510 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'58 2020-08-24T01:16:17.510 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'59 2020-08-24T01:16:17.510 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'60 2020-08-24T01:16:17.510 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'61 2020-08-24T01:16:17.510 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'62 2020-08-24T01:16:17.511 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'63 2020-08-24T01:16:17.511 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'64 2020-08-24T01:16:17.511 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'65 2020-08-24T01:16:17.511 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'66 2020-08-24T01:16:17.511 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'67 2020-08-24T01:16:17.512 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'68 2020-08-24T01:16:17.512 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'69 2020-08-24T01:16:17.512 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'70 2020-08-24T01:16:17.512 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'71 2020-08-24T01:16:17.512 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'72 2020-08-24T01:16:17.513 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'73 2020-08-24T01:16:17.513 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'74 2020-08-24T01:16:17.513 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'75 2020-08-24T01:16:17.513 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'76 2020-08-24T01:16:17.513 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'77 2020-08-24T01:16:17.514 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'78 2020-08-24T01:16:17.514 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'79 2020-08-24T01:16:17.514 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'80 2020-08-24T01:16:17.514 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'81 2020-08-24T01:16:17.514 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'82 2020-08-24T01:16:17.515 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'83 2020-08-24T01:16:17.515 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'84 2020-08-24T01:16:17.515 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'85 2020-08-24T01:16:17.515 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'86 2020-08-24T01:16:17.515 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'87 2020-08-24T01:16:17.515 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'88 2020-08-24T01:16:17.516 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'89 2020-08-24T01:16:17.516 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'90 2020-08-24T01:16:17.516 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'91 2020-08-24T01:16:17.516 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'92 2020-08-24T01:16:17.516 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'93 2020-08-24T01:16:17.517 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'94 2020-08-24T01:16:17.517 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'95 2020-08-24T01:16:17.517 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'96 2020-08-24T01:16:17.517 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'97 2020-08-24T01:16:17.517 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'98 2020-08-24T01:16:17.518 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'99 2020-08-24T01:16:17.518 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'100 2020-08-24T01:16:17.518 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-08-24T01:16:17.518 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-08-24T01:16:17.518 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-08-24T01:16:17.519 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-08-24T01:16:17.519 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-08-24T01:16:17.519 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-08-24T01:16:17.519 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-08-24T01:16:17.519 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-08-24T01:16:17.520 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-08-24T01:16:17.520 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-08-24T01:16:17.520 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-08-24T01:16:17.520 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-08-24T01:16:17.520 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-08-24T01:16:17.521 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-08-24T01:16:17.521 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-08-24T01:16:17.521 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-08-24T01:16:17.521 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-08-24T01:16:17.521 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-08-24T01:16:17.522 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-08-24T01:16:17.522 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-08-24T01:16:17.522 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-08-24T01:16:17.522 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-08-24T01:16:17.523 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-08-24T01:16:17.523 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-08-24T01:16:17.524 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-08-24T01:16:17.524 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-08-24T01:16:17.524 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-08-24T01:16:17.524 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-08-24T01:16:17.525 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-08-24T01:16:17.525 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-08-24T01:16:17.525 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-08-24T01:16:17.525 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-08-24T01:16:17.525 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-08-24T01:16:17.526 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-08-24T01:16:17.526 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-08-24T01:16:17.526 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-08-24T01:16:17.526 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-08-24T01:16:17.526 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-08-24T01:16:17.526 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-08-24T01:16:17.527 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-08-24T01:16:17.527 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-08-24T01:16:17.527 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-08-24T01:16:17.527 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-08-24T01:16:17.527 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-08-24T01:16:17.528 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-08-24T01:16:17.528 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-08-24T01:16:17.528 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-08-24T01:16:17.528 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-08-24T01:16:17.528 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to max 2 2020-08-24T01:16:17.529 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to copy log version 33'150 2020-08-24T01:16:17.529 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to copy log version 33'149 2020-08-24T01:16:17.529 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after earliest_dup_version 0 2020-08-24T01:16:17.529 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'1 2020-08-24T01:16:17.529 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'2 2020-08-24T01:16:17.530 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'3 2020-08-24T01:16:17.530 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'4 2020-08-24T01:16:17.530 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'5 2020-08-24T01:16:17.530 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'6 2020-08-24T01:16:17.530 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'7 2020-08-24T01:16:17.531 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'8 2020-08-24T01:16:17.531 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'9 2020-08-24T01:16:17.531 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'10 2020-08-24T01:16:17.531 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'11 2020-08-24T01:16:17.531 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'12 2020-08-24T01:16:17.532 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'13 2020-08-24T01:16:17.532 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'14 2020-08-24T01:16:17.532 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'15 2020-08-24T01:16:17.532 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'16 2020-08-24T01:16:17.532 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'17 2020-08-24T01:16:17.532 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'18 2020-08-24T01:16:17.533 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'19 2020-08-24T01:16:17.533 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'20 2020-08-24T01:16:17.533 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'21 2020-08-24T01:16:17.533 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'22 2020-08-24T01:16:17.534 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'23 2020-08-24T01:16:17.534 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'24 2020-08-24T01:16:17.534 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'25 2020-08-24T01:16:17.534 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'26 2020-08-24T01:16:17.534 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'27 2020-08-24T01:16:17.535 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'28 2020-08-24T01:16:17.535 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'29 2020-08-24T01:16:17.535 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'30 2020-08-24T01:16:17.535 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'31 2020-08-24T01:16:17.535 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'32 2020-08-24T01:16:17.535 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'33 2020-08-24T01:16:17.536 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'34 2020-08-24T01:16:17.536 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'35 2020-08-24T01:16:17.536 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'36 2020-08-24T01:16:17.536 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'37 2020-08-24T01:16:17.536 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'38 2020-08-24T01:16:17.537 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'39 2020-08-24T01:16:17.537 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'40 2020-08-24T01:16:17.537 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'41 2020-08-24T01:16:17.537 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'42 2020-08-24T01:16:17.537 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'43 2020-08-24T01:16:17.538 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'44 2020-08-24T01:16:17.538 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'45 2020-08-24T01:16:17.538 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'46 2020-08-24T01:16:17.538 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'47 2020-08-24T01:16:17.538 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'48 2020-08-24T01:16:17.539 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'49 2020-08-24T01:16:17.539 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'50 2020-08-24T01:16:17.539 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'51 2020-08-24T01:16:17.539 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'52 2020-08-24T01:16:17.540 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'53 2020-08-24T01:16:17.540 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'54 2020-08-24T01:16:17.540 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'55 2020-08-24T01:16:17.540 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'56 2020-08-24T01:16:17.540 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'57 2020-08-24T01:16:17.541 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'58 2020-08-24T01:16:17.541 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'59 2020-08-24T01:16:17.541 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'60 2020-08-24T01:16:17.541 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'61 2020-08-24T01:16:17.541 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'62 2020-08-24T01:16:17.543 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'63 2020-08-24T01:16:17.543 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'64 2020-08-24T01:16:17.543 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'65 2020-08-24T01:16:17.543 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'66 2020-08-24T01:16:17.543 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'67 2020-08-24T01:16:17.544 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'68 2020-08-24T01:16:17.544 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'69 2020-08-24T01:16:17.544 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'70 2020-08-24T01:16:17.544 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'71 2020-08-24T01:16:17.544 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'72 2020-08-24T01:16:17.545 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'73 2020-08-24T01:16:17.545 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'74 2020-08-24T01:16:17.545 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'75 2020-08-24T01:16:17.545 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'76 2020-08-24T01:16:17.546 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'77 2020-08-24T01:16:17.546 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'78 2020-08-24T01:16:17.546 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'79 2020-08-24T01:16:17.546 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'80 2020-08-24T01:16:17.546 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'81 2020-08-24T01:16:17.547 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'82 2020-08-24T01:16:17.547 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'83 2020-08-24T01:16:17.547 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'84 2020-08-24T01:16:17.547 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'85 2020-08-24T01:16:17.547 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'86 2020-08-24T01:16:17.547 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'87 2020-08-24T01:16:17.548 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'88 2020-08-24T01:16:17.548 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'89 2020-08-24T01:16:17.548 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'90 2020-08-24T01:16:17.548 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'91 2020-08-24T01:16:17.548 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'92 2020-08-24T01:16:17.549 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'93 2020-08-24T01:16:17.549 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'94 2020-08-24T01:16:17.549 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'95 2020-08-24T01:16:17.549 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'96 2020-08-24T01:16:17.549 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'97 2020-08-24T01:16:17.550 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'98 2020-08-24T01:16:17.550 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'99 2020-08-24T01:16:17.550 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'100 2020-08-24T01:16:17.550 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-08-24T01:16:17.550 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-08-24T01:16:17.551 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-08-24T01:16:17.551 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-08-24T01:16:17.551 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-08-24T01:16:17.551 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-08-24T01:16:17.551 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-08-24T01:16:17.552 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-08-24T01:16:17.552 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-08-24T01:16:17.552 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-08-24T01:16:17.552 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-08-24T01:16:17.552 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-08-24T01:16:17.553 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-08-24T01:16:17.553 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-08-24T01:16:17.553 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-08-24T01:16:17.553 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-08-24T01:16:17.554 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-08-24T01:16:17.554 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-08-24T01:16:17.554 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-08-24T01:16:17.554 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-08-24T01:16:17.554 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-08-24T01:16:17.555 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-08-24T01:16:17.555 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-08-24T01:16:17.555 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-08-24T01:16:17.555 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-08-24T01:16:17.555 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-08-24T01:16:17.556 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-08-24T01:16:17.556 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-08-24T01:16:17.565 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-08-24T01:16:17.566 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-08-24T01:16:17.566 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-08-24T01:16:17.566 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-08-24T01:16:17.566 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-08-24T01:16:17.567 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-08-24T01:16:17.567 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-08-24T01:16:17.567 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-08-24T01:16:17.567 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-08-24T01:16:17.567 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-08-24T01:16:17.568 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-08-24T01:16:17.568 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-08-24T01:16:17.568 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-08-24T01:16:17.568 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-08-24T01:16:17.568 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-08-24T01:16:17.568 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-08-24T01:16:17.569 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-08-24T01:16:17.569 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-08-24T01:16:17.569 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-08-24T01:16:17.569 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:15:27.713+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-08-24T01:16:17.570 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_after v 33'150 2020-08-24T01:16:17.570 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after earliest_dup_version 0 2020-08-24T01:16:17.570 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'1 2020-08-24T01:16:17.570 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'2 2020-08-24T01:16:17.570 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'3 2020-08-24T01:16:17.571 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'4 2020-08-24T01:16:17.571 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'5 2020-08-24T01:16:17.571 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'6 2020-08-24T01:16:17.571 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'7 2020-08-24T01:16:17.571 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'8 2020-08-24T01:16:17.572 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'9 2020-08-24T01:16:17.572 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'10 2020-08-24T01:16:17.572 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'11 2020-08-24T01:16:17.572 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'12 2020-08-24T01:16:17.572 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'13 2020-08-24T01:16:17.573 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'14 2020-08-24T01:16:17.573 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'15 2020-08-24T01:16:17.573 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'16 2020-08-24T01:16:17.573 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'17 2020-08-24T01:16:17.573 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'18 2020-08-24T01:16:17.574 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'19 2020-08-24T01:16:17.574 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'20 2020-08-24T01:16:17.574 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'21 2020-08-24T01:16:17.575 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'22 2020-08-24T01:16:17.575 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'23 2020-08-24T01:16:17.575 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'24 2020-08-24T01:16:17.575 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'25 2020-08-24T01:16:17.576 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'26 2020-08-24T01:16:17.576 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'27 2020-08-24T01:16:17.576 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'28 2020-08-24T01:16:17.576 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'29 2020-08-24T01:16:17.576 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'30 2020-08-24T01:16:17.577 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'31 2020-08-24T01:16:17.577 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'32 2020-08-24T01:16:17.577 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'33 2020-08-24T01:16:17.577 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'34 2020-08-24T01:16:17.577 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'35 2020-08-24T01:16:17.578 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'36 2020-08-24T01:16:17.578 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'37 2020-08-24T01:16:17.578 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'38 2020-08-24T01:16:17.578 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'39 2020-08-24T01:16:17.578 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'40 2020-08-24T01:16:17.579 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'41 2020-08-24T01:16:17.579 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'42 2020-08-24T01:16:17.579 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'43 2020-08-24T01:16:17.579 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'44 2020-08-24T01:16:17.579 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'45 2020-08-24T01:16:17.580 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'46 2020-08-24T01:16:17.580 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'47 2020-08-24T01:16:17.580 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'48 2020-08-24T01:16:17.580 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'49 2020-08-24T01:16:17.580 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'50 2020-08-24T01:16:17.581 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'51 2020-08-24T01:16:17.582 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'52 2020-08-24T01:16:17.582 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'53 2020-08-24T01:16:17.582 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'54 2020-08-24T01:16:17.583 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'55 2020-08-24T01:16:17.583 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'56 2020-08-24T01:16:17.583 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'57 2020-08-24T01:16:17.583 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'58 2020-08-24T01:16:17.584 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'59 2020-08-24T01:16:17.584 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'60 2020-08-24T01:16:17.584 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'61 2020-08-24T01:16:17.584 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'62 2020-08-24T01:16:17.585 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'63 2020-08-24T01:16:17.585 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'64 2020-08-24T01:16:17.585 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'65 2020-08-24T01:16:17.585 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'66 2020-08-24T01:16:17.586 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'67 2020-08-24T01:16:17.586 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'68 2020-08-24T01:16:17.586 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'69 2020-08-24T01:16:17.586 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'70 2020-08-24T01:16:17.587 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'71 2020-08-24T01:16:17.587 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'72 2020-08-24T01:16:17.587 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'73 2020-08-24T01:16:17.587 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'74 2020-08-24T01:16:17.588 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'75 2020-08-24T01:16:17.588 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'76 2020-08-24T01:16:17.588 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'77 2020-08-24T01:16:17.588 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'78 2020-08-24T01:16:17.588 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'79 2020-08-24T01:16:17.589 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'80 2020-08-24T01:16:17.589 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'81 2020-08-24T01:16:17.589 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'82 2020-08-24T01:16:17.589 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'83 2020-08-24T01:16:17.590 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'84 2020-08-24T01:16:17.590 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'85 2020-08-24T01:16:17.590 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'86 2020-08-24T01:16:17.590 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'87 2020-08-24T01:16:17.591 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'88 2020-08-24T01:16:17.591 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'89 2020-08-24T01:16:17.591 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'90 2020-08-24T01:16:17.591 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'91 2020-08-24T01:16:17.591 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'92 2020-08-24T01:16:17.592 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'93 2020-08-24T01:16:17.592 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'94 2020-08-24T01:16:17.592 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'95 2020-08-24T01:16:17.592 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'96 2020-08-24T01:16:17.593 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'97 2020-08-24T01:16:17.593 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'98 2020-08-24T01:16:17.593 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'99 2020-08-24T01:16:17.593 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup version 33'100 2020-08-24T01:16:17.593 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'101 2020-08-24T01:16:17.594 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'102 2020-08-24T01:16:17.594 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'103 2020-08-24T01:16:17.594 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'104 2020-08-24T01:16:17.594 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'105 2020-08-24T01:16:17.595 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'106 2020-08-24T01:16:17.595 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'107 2020-08-24T01:16:17.595 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'108 2020-08-24T01:16:17.595 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'109 2020-08-24T01:16:17.596 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'110 2020-08-24T01:16:17.596 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'111 2020-08-24T01:16:17.596 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'112 2020-08-24T01:16:17.596 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'113 2020-08-24T01:16:17.597 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'114 2020-08-24T01:16:17.597 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'115 2020-08-24T01:16:17.597 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'116 2020-08-24T01:16:17.597 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'117 2020-08-24T01:16:17.598 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'118 2020-08-24T01:16:17.598 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'119 2020-08-24T01:16:17.599 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'120 2020-08-24T01:16:17.599 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'121 2020-08-24T01:16:17.599 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'122 2020-08-24T01:16:17.599 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'123 2020-08-24T01:16:17.600 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'124 2020-08-24T01:16:17.600 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'125 2020-08-24T01:16:17.600 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'126 2020-08-24T01:16:17.600 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'127 2020-08-24T01:16:17.601 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'128 2020-08-24T01:16:17.601 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'129 2020-08-24T01:16:17.601 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'130 2020-08-24T01:16:17.601 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'131 2020-08-24T01:16:17.602 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'132 2020-08-24T01:16:17.602 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'133 2020-08-24T01:16:17.602 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'134 2020-08-24T01:16:17.603 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'135 2020-08-24T01:16:17.603 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'136 2020-08-24T01:16:17.603 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'137 2020-08-24T01:16:17.603 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'138 2020-08-24T01:16:17.604 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'139 2020-08-24T01:16:17.604 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'140 2020-08-24T01:16:17.604 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'141 2020-08-24T01:16:17.604 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'142 2020-08-24T01:16:17.605 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'143 2020-08-24T01:16:17.605 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'144 2020-08-24T01:16:17.605 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'145 2020-08-24T01:16:17.606 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'146 2020-08-24T01:16:17.606 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'147 2020-08-24T01:16:17.607 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'148 2020-08-24T01:16:17.607 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'149 2020-08-24T01:16:17.607 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:16:05.341+0000 7fdcda069700 20 copy_up_to/copy_after copy dup from log version 33'150 2020-08-24T01:16:17.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-08-24T01:16:17.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2020-08-24T01:16:17.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-08-24T01:16:17.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:17.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:16:17.608 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T01:16:17.609 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:16:17.609 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:16:17.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:16:17.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:16:17.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:16:17.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:16:17.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:16:17.610 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:16:17.611 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:16:17.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:16:17.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:16:17.611 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:16:17.611 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:16:17.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:16:17.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:16:17.612 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:16:17.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:16:17.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:16:17.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T01:16:17.613 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:16:17.613 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:17.613 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:17.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T01:16:17.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:16:17.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:16:17.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-08-24T01:16:17.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-08-24T01:16:17.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:17.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-08-24T01:16:17.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:17.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:16:17.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T01:16:17.616 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:16:17.616 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:16:17.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:16:17.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:16:17.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:16:17.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:16:17.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:16:17.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:16:17.617 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:16:17.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:16:17.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:16:17.618 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:16:17.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:16:17.618 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:16:17.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:16:17.619 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:16:17.619 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:16:17.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:16:17.620 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T01:16:17.620 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:16:17.620 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:17.620 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:17.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T01:16:17.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:16:17.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:16:17.621 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-08-24T01:16:17.621 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:16:17.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:17.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:17.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71396 2020-08-24T01:16:17.622 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:16:17.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-08-24T01:16:17.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:16:17.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-08-24T01:16:17.623 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_1 td/osd-backfill-recovery-log 2020-08-24T01:16:17.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:121: TEST_recovery_1: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:17.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:123: TEST_recovery_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 40 0 40 2020-08-24T01:16:17.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:17.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:17.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=40 2020-08-24T01:16:17.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=0 2020-08-24T01:16:17.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=40 2020-08-24T01:16:17.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-08-24T01:16:17.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-08-24T01:16:17.625 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-08-24T01:16:17.626 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:17.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:16:17.627 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:16:17.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:16:17.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-08-24T01:16:17.628 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-08-24T01:16:17.855 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:16:17.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:17.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:17.857 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:17.857 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:17.857 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:17.857 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:17.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:16:18.044 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:16:18.045 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:16:18.046 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:16:18.046 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:16:18.046 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:16:18.047 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:16:18.047 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:16:18.047 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:16:18.048 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:16:18.048 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:16:18.049 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:18.049 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:18.049 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:16:18.050 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:16:18.050 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get fsid 2020-08-24T01:16:18.147 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:16:18.147 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:16:18.147 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:16:18.148 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:16:18.149 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:16:18.149 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:16:18.149 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:16:18.149 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:16:18.150 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:16:18.150 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:18.150 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:18.150 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:16:18.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:16:18.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get mon_host 2020-08-24T01:16:18.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-08-24T01:16:18.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:18.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:16:18.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:16:18.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:16:18.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-08-24T01:16:18.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:16:18.816 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:16:18.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:18.817 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:18.818 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:18.818 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:18.818 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:18.818 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:18.819 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:16:18.819 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:16:18.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-08-24T01:16:18.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:18.852 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:18.852 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-08-24T01:16:18.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-08-24T01:16:18.852 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:16:18.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-08-24T01:16:18.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:18.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:16:18.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:16:18.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:16:18.853 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-08-24T01:16:18.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:16:18.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:16:18.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:16:18.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:16:18.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-08-24T01:16:18.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-08-24T01:16:18.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:16:18.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:18.855 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:16:18.855 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:16:18.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:18.856 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:18.858 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:18.858 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:18.858 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:18.858 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:18.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:18.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:16:18.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:16:18.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:16:18.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:16:18.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:16:18.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:16:18.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:16:18.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:18.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:16:18.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:16:18.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-08-24T01:16:18.880 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:16:18.880 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0e92051b-77ee-43ed-a84a-a30447a742b9 2020-08-24T01:16:18.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 0e92051b-77ee-43ed-a84a-a30447a742b9' 2020-08-24T01:16:18.881 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:16:18.881 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 0e92051b-77ee-43ed-a84a-a30447a742b9 2020-08-24T01:16:18.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDiFENfTXQPNBAA/9frR3x2lV67TnkXIKJBcA== 2020-08-24T01:16:18.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDiFENfTXQPNBAA/9frR3x2lV67TnkXIKJBcA=="}' 2020-08-24T01:16:18.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0e92051b-77ee-43ed-a84a-a30447a742b9 -i td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:16:19.210 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:16:19.220 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:16:19.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDiFENfTXQPNBAA/9frR3x2lV67TnkXIKJBcA== --osd-uuid 0e92051b-77ee-43ed-a84a-a30447a742b9 2020-08-24T01:16:19.253 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.249+0000 7f46095dfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:19.255 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.249+0000 7f46095dfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:19.256 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.253+0000 7f46095dfd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:19.258 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.253+0000 7f46095dfd80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:16:19.258 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.253+0000 7f46095dfd80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:16:19.258 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.253+0000 7f46095dfd80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:16:19.301 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.297+0000 7f46095dfd80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-08-24T01:16:19.644 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:19.641+0000 7f46095dfd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:16:22.390 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-08-24T01:16:22.391 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:16:22.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:16:22.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:16:22.392 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:16:22.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:16:22.850 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:16:22.850 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:16:22.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:22.851 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:16:22.852 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:16:22.881 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:22.877+0000 7fc640e57d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:22.883 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:22.877+0000 7fc640e57d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:22.884 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:22.881+0000 7fc640e57d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:23.231 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:16:23.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:16:23.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:16:23.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:16:23.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:16:23.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:23.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:16:23.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:23.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:16:23.233 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:16:23.430 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:23.425+0000 7fc640e57d80 -1 Falling back to public interface 2020-08-24T01:16:23.583 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:16:24.214 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:24.205+0000 7fc640e57d80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:16:24.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:16:24.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:24.585 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:16:24.586 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:16:24.586 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:24.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:16:24.957 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/90366,v1:127.0.0.1:6803/90366] [v2:127.0.0.1:6804/90366,v1:127.0.0.1:6805/90366] exists,up 0e92051b-77ee-43ed-a84a-a30447a742b9 2020-08-24T01:16:24.964 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:16:24.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:16:24.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:16:24.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:16:24.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-08-24T01:16:24.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:24.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:16:24.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:16:24.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:16:24.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-08-24T01:16:24.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:16:24.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:16:24.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:16:24.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:16:24.969 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-08-24T01:16:24.970 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-08-24T01:16:24.970 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:16:24.970 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:24.970 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:16:24.970 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:16:24.971 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:24.971 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:24.971 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:24.971 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:24.971 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:24.973 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:24.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:24.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:16:24.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:16:24.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:16:24.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:16:24.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:16:24.975 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:16:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:16:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:16:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:16:24.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-08-24T01:16:24.977 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:16:24.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8b239007-cb39-4fbc-8e33-75af94f88e84 2020-08-24T01:16:24.977 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 8b239007-cb39-4fbc-8e33-75af94f88e84' 2020-08-24T01:16:24.977 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:16:24.978 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 8b239007-cb39-4fbc-8e33-75af94f88e84 2020-08-24T01:16:24.981 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDoFENfQBxqOhAA7Qcwx2E+idLKIK5hUsA6iw== 2020-08-24T01:16:24.981 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDoFENfQBxqOhAA7Qcwx2E+idLKIK5hUsA6iw=="}' 2020-08-24T01:16:24.982 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8b239007-cb39-4fbc-8e33-75af94f88e84 -i td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:16:25.530 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:16:25.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:16:25.540 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDoFENfQBxqOhAA7Qcwx2E+idLKIK5hUsA6iw== --osd-uuid 8b239007-cb39-4fbc-8e33-75af94f88e84 2020-08-24T01:16:25.568 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:25.561+0000 7fa2bb12ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:25.571 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:25.565+0000 7fa2bb12ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:25.571 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:25.569+0000 7fa2bb12ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:25.572 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:25.569+0000 7fa2bb12ed80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:16:25.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:25.569+0000 7fa2bb12ed80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:16:25.573 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:25.569+0000 7fa2bb12ed80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:16:25.658 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:25.653+0000 7fa2bb12ed80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-08-24T01:16:26.043 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:26.037+0000 7fa2bb12ed80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:16:28.736 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-08-24T01:16:28.737 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:16:28.738 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:16:28.739 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:16:28.739 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:16:29.201 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:16:29.201 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:16:29.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:29.202 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:16:29.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:16:29.203 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:16:29.232 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:29.229+0000 7f178e951d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:29.234 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:29.229+0000 7f178e951d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:29.236 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:29.233+0000 7f178e951d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:29.551 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:16:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:16:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:16:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:16:29.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:16:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:16:29.553 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:29.553 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:16:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:16:29.778 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:29.773+0000 7f178e951d80 -1 Falling back to public interface 2020-08-24T01:16:29.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:16:30.578 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:30.573+0000 7f178e951d80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:16:30.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:16:30.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:30.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:16:30.901 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:16:30.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:30.902 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:16:31.297 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/90721,v1:127.0.0.1:6811/90721] [v2:127.0.0.1:6812/90721,v1:127.0.0.1:6813/90721] exists,up 8b239007-cb39-4fbc-8e33-75af94f88e84 2020-08-24T01:16:31.298 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 01dc779d-6e7a-4ba4-b68f-2aab31d69e93 2020-08-24T01:16:31.298 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:16:31.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:16:31.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:16:31.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:16:31.300 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-08-24T01:16:31.300 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:31.300 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:16:31.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T01:16:31.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:16:31.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-08-24T01:16:31.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:16:31.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:16:31.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:16:31.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:16:31.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-08-24T01:16:31.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-08-24T01:16:31.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:16:31.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:31.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:16:31.305 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:16:31.305 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:31.305 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:31.314 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:31.314 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:31.314 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:31.315 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:31.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:31.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:16:31.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:16:31.316 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:16:31.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:16:31.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:16:31.317 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:16:31.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:16:31.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:31.318 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:16:31.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:16:31.319 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-08-24T01:16:31.319 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:16:31.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=01dc779d-6e7a-4ba4-b68f-2aab31d69e93 2020-08-24T01:16:31.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 01dc779d-6e7a-4ba4-b68f-2aab31d69e93' 2020-08-24T01:16:31.320 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:16:31.328 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDvFENfD6K/EhAA7ovZr7JvwYJS0nhExnPPKQ== 2020-08-24T01:16:31.328 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDvFENfD6K/EhAA7ovZr7JvwYJS0nhExnPPKQ=="}' 2020-08-24T01:16:31.329 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 01dc779d-6e7a-4ba4-b68f-2aab31d69e93 -i td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:16:31.851 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:16:31.868 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:16:31.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDvFENfD6K/EhAA7ovZr7JvwYJS0nhExnPPKQ== --osd-uuid 01dc779d-6e7a-4ba4-b68f-2aab31d69e93 2020-08-24T01:16:31.904 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:31.897+0000 7fbb858e8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:31.925 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:31.921+0000 7fbb858e8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:31.927 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:31.921+0000 7fbb858e8d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:31.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:31.925+0000 7fbb858e8d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:16:31.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:31.925+0000 7fbb858e8d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:16:31.928 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:31.925+0000 7fbb858e8d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:16:31.982 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:31.977+0000 7fbb858e8d80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-08-24T01:16:32.317 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:32.313+0000 7fbb858e8d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:16:35.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-08-24T01:16:35.180 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:16:35.182 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T01:16:35.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T01:16:35.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:16:35.664 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:16:35.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T01:16:35.664 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:35.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:16:35.665 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:16:35.666 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:16:35.691 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:35.685+0000 7f07163fad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:35.694 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:35.689+0000 7f07163fad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:35.695 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:35.689+0000 7f07163fad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:36.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T01:16:36.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:16:36.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:16:36.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:16:36.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:16:36.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:36.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:16:36.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:36.023 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:16:36.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:16:36.233 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:36.229+0000 7f07163fad80 -1 Falling back to public interface 2020-08-24T01:16:36.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:16:37.060 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:37.053+0000 7f07163fad80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T01:16:37.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:16:37.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:37.427 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:16:37.427 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:16:37.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:37.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:16:37.825 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/91076,v1:127.0.0.1:6819/91076] [v2:127.0.0.1:6820/91076,v1:127.0.0.1:6821/91076] exists,up 01dc779d-6e7a-4ba4-b68f-2aab31d69e93 2020-08-24T01:16:37.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:16:37.825 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:16:37.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:16:37.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:16:37.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-08-24T01:16:37.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:37.826 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:16:37.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-08-24T01:16:37.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:16:37.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-08-24T01:16:37.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:16:37.827 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:16:37.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:16:37.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:16:37.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-08-24T01:16:37.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-08-24T01:16:37.828 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:16:37.829 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:37.829 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:16:37.829 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:16:37.829 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:37.829 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:37.832 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:37.832 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:37.833 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:37.833 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:37.833 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:37.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:16:37.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:16:37.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:16:37.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:16:37.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:16:37.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:16:37.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:16:37.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:37.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:16:37.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:16:37.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-08-24T01:16:37.836 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:16:37.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=4a2af660-ec42-4898-99d8-3447ca053e7f 2020-08-24T01:16:37.836 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 4a2af660-ec42-4898-99d8-3447ca053e7f' 2020-08-24T01:16:37.837 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:16:37.837 INFO:tasks.workunit.client.0.smithi060.stdout:add osd3 4a2af660-ec42-4898-99d8-3447ca053e7f 2020-08-24T01:16:37.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD1FENf2ow/MhAAHpAZ5jyszuVStafX+n6iJQ== 2020-08-24T01:16:37.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD1FENf2ow/MhAAHpAZ5jyszuVStafX+n6iJQ=="}' 2020-08-24T01:16:37.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 4a2af660-ec42-4898-99d8-3447ca053e7f -i td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:16:38.433 INFO:tasks.workunit.client.0.smithi060.stdout:3 2020-08-24T01:16:38.445 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:16:38.446 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1FENf2ow/MhAAHpAZ5jyszuVStafX+n6iJQ== --osd-uuid 4a2af660-ec42-4898-99d8-3447ca053e7f 2020-08-24T01:16:38.480 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.477+0000 7fdf28db5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:38.483 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.477+0000 7fdf28db5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:38.484 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.481+0000 7fdf28db5d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:38.485 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.481+0000 7fdf28db5d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:16:38.485 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.481+0000 7fdf28db5d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:16:38.486 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.481+0000 7fdf28db5d80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:16:38.537 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.533+0000 7fdf28db5d80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-08-24T01:16:38.897 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:38.893+0000 7fdf28db5d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:16:41.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-08-24T01:16:41.668 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:16:41.670 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd3 key to auth repository 2020-08-24T01:16:41.670 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-08-24T01:16:41.670 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:16:42.135 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.3 2020-08-24T01:16:42.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-08-24T01:16:42.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:16:42.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:42.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:16:42.137 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:16:42.166 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:42.160+0000 7f2d9a4e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:42.169 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:42.164+0000 7f2d9a4e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:42.171 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:42.164+0000 7f2d9a4e9d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:42.482 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-08-24T01:16:42.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:16:42.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-08-24T01:16:42.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:16:42.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:16:42.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:42.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:16:42.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:42.484 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:16:42.485 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:16:42.722 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:42.716+0000 7f2d9a4e9d80 -1 Falling back to public interface 2020-08-24T01:16:42.859 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:16:43.532 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:43.520+0000 7f2d9a4e9d80 -1 osd.3 0 log_to_monitors {default=true} 2020-08-24T01:16:43.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:16:43.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:43.862 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:16:43.862 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:16:43.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:43.863 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:16:44.232 INFO:tasks.workunit.client.0.smithi060.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/91431,v1:127.0.0.1:6827/91431] [v2:127.0.0.1:6828/91431,v1:127.0.0.1:6829/91431] exists,up 4a2af660-ec42-4898-99d8-3447ca053e7f 2020-08-24T01:16:44.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:16:44.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:16:44.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:16:44.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:16:44.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-08-24T01:16:44.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:44.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:16:44.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-08-24T01:16:44.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:16:44.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-08-24T01:16:44.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:16:44.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:16:44.235 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:16:44.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:16:44.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-08-24T01:16:44.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-08-24T01:16:44.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:16:44.236 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:44.237 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:16:44.237 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:16:44.237 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:44.237 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:44.239 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:44.239 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:44.240 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:44.240 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:44.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:44.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:16:44.240 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:16:44.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:16:44.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:16:44.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:16:44.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:16:44.241 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:16:44.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:44.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:16:44.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:16:44.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-08-24T01:16:44.243 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:16:44.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=3e33aef3-e130-48f4-8e9b-83820c89b9e9 2020-08-24T01:16:44.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 3e33aef3-e130-48f4-8e9b-83820c89b9e9' 2020-08-24T01:16:44.244 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:16:44.244 INFO:tasks.workunit.client.0.smithi060.stdout:add osd4 3e33aef3-e130-48f4-8e9b-83820c89b9e9 2020-08-24T01:16:44.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD8FENfujZSDxAA26nJqu8RDOTzNE1wNdu2+g== 2020-08-24T01:16:44.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD8FENfujZSDxAA26nJqu8RDOTzNE1wNdu2+g=="}' 2020-08-24T01:16:44.258 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 3e33aef3-e130-48f4-8e9b-83820c89b9e9 -i td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:16:44.824 INFO:tasks.workunit.client.0.smithi060.stdout:4 2020-08-24T01:16:44.841 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:16:44.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD8FENfujZSDxAA26nJqu8RDOTzNE1wNdu2+g== --osd-uuid 3e33aef3-e130-48f4-8e9b-83820c89b9e9 2020-08-24T01:16:44.876 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:44.868+0000 7f11afc35d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:44.878 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:44.872+0000 7f11afc35d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:44.879 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:44.872+0000 7f11afc35d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:44.880 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:44.876+0000 7f11afc35d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:16:44.880 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:44.876+0000 7f11afc35d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:16:44.880 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:44.876+0000 7f11afc35d80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:16:44.939 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:44.932+0000 7f11afc35d80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-08-24T01:16:45.340 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:45.332+0000 7f11afc35d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:16:48.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-08-24T01:16:48.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:16:48.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-08-24T01:16:48.131 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:16:48.132 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd4 key to auth repository 2020-08-24T01:16:48.600 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-08-24T01:16:48.601 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.4 2020-08-24T01:16:48.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:48.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:16:48.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:16:48.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:16:48.633 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:48.628+0000 7f2cea416d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:48.636 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:48.632+0000 7f2cea416d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:48.638 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:48.632+0000 7f2cea416d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:48.972 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-08-24T01:16:48.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:16:48.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-08-24T01:16:48.973 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:16:48.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:16:48.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:48.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:16:48.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:48.974 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:16:48.975 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:16:49.170 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:49.164+0000 7f2cea416d80 -1 Falling back to public interface 2020-08-24T01:16:49.364 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:16:49.951 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:49.944+0000 7f2cea416d80 -1 osd.4 0 log_to_monitors {default=true} 2020-08-24T01:16:50.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:16:50.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:50.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:16:50.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:50.368 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:16:50.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:16:50.783 INFO:tasks.workunit.client.0.smithi060.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/91786,v1:127.0.0.1:6835/91786] [v2:127.0.0.1:6836/91786,v1:127.0.0.1:6837/91786] exists,up 3e33aef3-e130-48f4-8e9b-83820c89b9e9 2020-08-24T01:16:50.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:16:50.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:16:50.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:16:50.784 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:16:50.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-08-24T01:16:50.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:16:50.785 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:16:50.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-08-24T01:16:50.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:16:50.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-08-24T01:16:50.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:16:50.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:16:50.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:16:50.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:16:50.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-08-24T01:16:50.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-08-24T01:16:50.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:16:50.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10' 2020-08-24T01:16:50.789 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:16:50.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:16:50.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:16:50.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:16:50.791 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:16:50.791 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:16:50.791 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:16:50.792 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:50.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:16:50.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:16:50.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:16:50.803 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:16:50.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:16:50.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:16:50.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:16:50.804 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:16:50.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:50.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:16:50.806 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:16:50.806 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-08-24T01:16:50.807 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:16:50.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=a91af690-a1e1-46f1-9c91-7812b6088650 2020-08-24T01:16:50.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 a91af690-a1e1-46f1-9c91-7812b6088650' 2020-08-24T01:16:50.808 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:16:50.809 INFO:tasks.workunit.client.0.smithi060.stdout:add osd5 a91af690-a1e1-46f1-9c91-7812b6088650 2020-08-24T01:16:50.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQACFUNf3NsJMBAAJQnhi4DnWwEVNlPL2IKllg== 2020-08-24T01:16:50.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQACFUNf3NsJMBAAJQnhi4DnWwEVNlPL2IKllg=="}' 2020-08-24T01:16:50.811 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new a91af690-a1e1-46f1-9c91-7812b6088650 -i td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:16:51.319 INFO:tasks.workunit.client.0.smithi060.stdout:5 2020-08-24T01:16:51.336 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:16:51.338 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQACFUNf3NsJMBAAJQnhi4DnWwEVNlPL2IKllg== --osd-uuid a91af690-a1e1-46f1-9c91-7812b6088650 2020-08-24T01:16:51.372 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.364+0000 7fba58f06d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:51.374 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.368+0000 7fba58f06d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:51.377 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.368+0000 7fba58f06d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:51.377 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.368+0000 7fba58f06d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:16:51.378 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.368+0000 7fba58f06d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:16:51.378 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.368+0000 7fba58f06d80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:16:51.422 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.412+0000 7fba58f06d80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-08-24T01:16:51.791 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:51.784+0000 7fba58f06d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:16:54.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-08-24T01:16:54.764 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:16:54.765 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd5 key to auth repository 2020-08-24T01:16:54.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-08-24T01:16:54.766 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:16:55.243 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.5 2020-08-24T01:16:55.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-08-24T01:16:55.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:16:55.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:16:55.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:16:55.245 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:16:55.274 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:55.268+0000 7fa219df2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:55.277 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:55.272+0000 7fa219df2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:55.278 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:55.272+0000 7fa219df2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:16:55.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-08-24T01:16:55.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:16:55.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-08-24T01:16:55.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:16:55.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:16:55.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:55.604 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:16:55.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:55.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:16:55.605 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:16:55.818 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:55.812+0000 7fa219df2d80 -1 Falling back to public interface 2020-08-24T01:16:55.976 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:16:56.626 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:16:56.620+0000 7fa219df2d80 -1 osd.5 0 log_to_monitors {default=true} 2020-08-24T01:16:56.978 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:16:56.979 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:56.979 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:16:56.979 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:16:56.980 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:56.980 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:16:57.328 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:16:58.330 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:16:58.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:16:58.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:16:58.331 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:16:58.332 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:16:58.332 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:16:58.695 INFO:tasks.workunit.client.0.smithi060.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/92141,v1:127.0.0.1:6843/92141] [v2:127.0.0.1:6844/92141,v1:127.0.0.1:6845/92141] exists,up a91af690-a1e1-46f1-9c91-7812b6088650 2020-08-24T01:16:58.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:16:58.695 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:16:58.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:16:58.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-08-24T01:16:58.696 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-08-24T01:16:59.237 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T01:16:59.254 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:17:00.247 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 40 2020-08-24T01:17:00.248 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:00.249 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-08-24T01:17:00.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:00.523 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-08-24T01:17:00.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:00.701 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-08-24T01:17:00.888 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:00.889 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-08-24T01:17:01.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:01.086 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-08-24T01:17:01.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:01.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-08-24T01:17:01.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:01.411 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-08-24T01:17:01.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:01.552 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-08-24T01:17:01.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:01.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-08-24T01:17:01.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:01.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-08-24T01:17:02.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:02.076 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-08-24T01:17:02.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:02.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-08-24T01:17:02.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:02.407 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-08-24T01:17:02.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:02.569 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-08-24T01:17:02.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:02.728 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-08-24T01:17:02.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:02.883 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-08-24T01:17:03.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:03.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-08-24T01:17:03.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:03.244 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-08-24T01:17:03.428 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:03.429 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-08-24T01:17:03.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:03.579 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-08-24T01:17:03.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:03.757 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-08-24T01:17:03.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:03.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-08-24T01:17:04.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:04.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-08-24T01:17:04.242 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:04.243 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-08-24T01:17:04.409 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:04.410 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-08-24T01:17:04.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:04.576 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-08-24T01:17:04.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:04.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-08-24T01:17:04.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:04.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-08-24T01:17:05.016 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:05.017 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-08-24T01:17:05.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:05.175 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-08-24T01:17:05.333 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:05.333 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-08-24T01:17:05.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:05.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-08-24T01:17:05.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:05.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-08-24T01:17:05.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:05.810 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-08-24T01:17:05.957 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:05.957 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-08-24T01:17:06.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:06.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-08-24T01:17:06.262 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:06.263 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-08-24T01:17:06.439 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:06.440 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-08-24T01:17:06.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:06.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-08-24T01:17:06.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:17:06.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-08-24T01:17:06.954 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-08-24T01:17:06.954 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-08-24T01:17:07.290 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:17:07.300 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-08-24T01:17:07.822 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-08-24T01:17:07.831 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-08-24T01:17:07.832 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-08-24T01:17:08.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-08-24T01:17:08.834 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-08-24T01:17:08.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-08-24T01:17:08.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-08-24T01:17:08.835 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-08-24T01:17:08.836 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-08-24T01:17:08.836 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-08-24T01:17:08.836 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-08-24T01:17:08.837 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-08-24T01:17:08.837 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-08-24T01:17:08.837 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-08-24T01:17:09.039 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-08-24T01:17:09.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-08-24T01:17:09.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-08-24T01:17:09.040 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:17:09.040 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:17:09.406 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:17:09.407 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:17:09.407 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:17:09.407 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:17:09.407 INFO:tasks.workunit.client.0.smithi060.stderr:4 2020-08-24T01:17:09.407 INFO:tasks.workunit.client.0.smithi060.stderr:5' 2020-08-24T01:17:09.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:17:09.408 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:09.408 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:17:09.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836491 2020-08-24T01:17:09.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836491 2020-08-24T01:17:09.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836491' 2020-08-24T01:17:09.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:09.564 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:17:09.722 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672970 2020-08-24T01:17:09.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672970 2020-08-24T01:17:09.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2020-08-24T01:17:09.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:09.724 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:17:09.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509449 2020-08-24T01:17:09.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509449 2020-08-24T01:17:09.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449' 2020-08-24T01:17:09.871 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:09.872 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:17:10.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345928 2020-08-24T01:17:10.022 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 85899345928 2020-08-24T01:17:10.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345928' 2020-08-24T01:17:10.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:10.023 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:17:10.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182406 2020-08-24T01:17:10.178 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 107374182406 2020-08-24T01:17:10.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345928 4-107374182406' 2020-08-24T01:17:10.179 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:10.179 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-08-24T01:17:10.336 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018885 2020-08-24T01:17:10.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 128849018885 2020-08-24T01:17:10.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-64424509449 3-85899345928 4-107374182406 5-128849018885' 2020-08-24T01:17:10.337 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:10.338 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836491 2020-08-24T01:17:10.338 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:10.339 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:17:10.340 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:10.340 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836491 2020-08-24T01:17:10.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836491 2020-08-24T01:17:10.341 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2020-08-24T01:17:10.341 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836491 2020-08-24T01:17:10.342 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:17:10.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836491 -lt 21474836491 2020-08-24T01:17:10.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:10.716 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672970 2020-08-24T01:17:10.716 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:10.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:17:10.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672970 2020-08-24T01:17:10.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:10.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672970 2020-08-24T01:17:10.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2020-08-24T01:17:10.720 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672970 2020-08-24T01:17:10.721 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:17:11.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672970 -lt 42949672970 2020-08-24T01:17:11.085 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:11.086 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509449 2020-08-24T01:17:11.086 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:11.087 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:17:11.089 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509449 2020-08-24T01:17:11.089 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:11.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509449 2020-08-24T01:17:11.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509449' 2020-08-24T01:17:11.092 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:17:11.092 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509449 2020-08-24T01:17:11.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509449 -lt 64424509449 2020-08-24T01:17:11.489 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:11.489 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-85899345928 2020-08-24T01:17:11.490 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:11.492 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:17:11.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345928 2020-08-24T01:17:11.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:11.494 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=85899345928 2020-08-24T01:17:11.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 85899345928' 2020-08-24T01:17:11.495 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 85899345928 2020-08-24T01:17:11.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:17:11.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 85899345928 -lt 85899345928 2020-08-24T01:17:11.848 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:11.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-107374182406 2020-08-24T01:17:11.849 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:11.852 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:17:11.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182406 2020-08-24T01:17:11.852 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:11.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=107374182406 2020-08-24T01:17:11.854 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 107374182406' 2020-08-24T01:17:11.854 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 107374182406 2020-08-24T01:17:11.855 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:17:12.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 107374182406 -lt 107374182406 2020-08-24T01:17:12.204 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:12.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 5-128849018885 2020-08-24T01:17:12.205 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:12.206 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=5 2020-08-24T01:17:12.207 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018885 2020-08-24T01:17:12.207 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:12.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=128849018885 2020-08-24T01:17:12.209 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.5 seq 128849018885' 2020-08-24T01:17:12.209 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 5 2020-08-24T01:17:12.210 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.5 seq 128849018885 2020-08-24T01:17:12.598 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 128849018885 -lt 128849018885 2020-08-24T01:17:12.598 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-08-24T01:17:12.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:12.599 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:13.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-08-24T01:17:13.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:13.052 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:13.052 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:13.052 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:13.052 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:13.052 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:13.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:13.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:17:13.394 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:13.395 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:13.395 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:13.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:17:13.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-08-24T01:17:13.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-08-24T01:17:13.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-08-24T01:17:13.846 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:17:13.847 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:17:13.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:17:13.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:13.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:13.947 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:13.948 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:13.948 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:13.948 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:13.948 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:14.299 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:17:14.300 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:14.301 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:14.301 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:14.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:17:14.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:17:14.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:17:14.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:17:14.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:17:14.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:17:14.795 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:17:14.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:17:14.796 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:17:15.259 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=7586 2020-08-24T01:17:15.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 7586 '!=' null 2020-08-24T01:17:15.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:17:15.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:17:15.260 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:17:15.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:17:15.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:15.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:15.359 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:15.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:15.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:15.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:15.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:15.709 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:17:15.709 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:15.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:15.710 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:16.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:17:16.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:17:16.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:17:16.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:17:16.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:17:16.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:17:16.161 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:17:16.161 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:17:16.162 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:17:16.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23762 2020-08-24T01:17:16.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23762 '!=' null 2020-08-24T01:17:16.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:17:16.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:17:16.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:17:16.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:17:16.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:16.716 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:16.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:16.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:16.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:16.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:16.718 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:17.069 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:17:17.070 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:17.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:17.071 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:17.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:17:17.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:17:17.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:17:17.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:17:17.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:17:17.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:17:17.521 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:17:17.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:17:17.522 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:17:17.980 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23762 2020-08-24T01:17:17.980 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23762 '!=' null 2020-08-24T01:17:17.980 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:17:17.980 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:17:17.981 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:17:18.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:17:18.082 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:18.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:18.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:18.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:18.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:18.083 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:18.084 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:18.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:17:18.427 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:18.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:18.428 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:18.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:17:18.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:17:18.873 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:17:18.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:17:18.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:17:18.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:17:18.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:17:18.875 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:17:18.875 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:17:19.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20299 2020-08-24T01:17:19.353 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20299 '!=' null 2020-08-24T01:17:19.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:17:19.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:17:19.354 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:17:19.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:17:19.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:19.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:19.456 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:19.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:19.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:19.457 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:19.458 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:19.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:17:19.806 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:19.806 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:19.806 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:20.266 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:17:20.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:17:20.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:17:20.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:17:20.267 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:17:20.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:17:20.268 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:17:20.268 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:17:20.269 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:17:20.705 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17072 2020-08-24T01:17:20.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17072 '!=' null 2020-08-24T01:17:20.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:17:20.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:17:20.706 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:17:20.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:17:20.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:20.808 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:20.808 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:20.808 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:20.809 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:20.809 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:20.809 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:21.164 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-08-24T01:17:21.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:21.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:21.165 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:21.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-08-24T01:17:21.612 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-08-24T01:17:21.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-08-24T01:17:21.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-08-24T01:17:21.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-08-24T01:17:21.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-08-24T01:17:21.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-08-24T01:17:21.614 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-08-24T01:17:21.615 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-08-24T01:17:22.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17072 2020-08-24T01:17:22.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17072 '!=' null 2020-08-24T01:17:22.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-08-24T01:17:22.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-08-24T01:17:22.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-08-24T01:17:22.150 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-08-24T01:17:22.150 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-08-24T01:17:22.151 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-08-24T01:17:22.152 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-08-24T01:17:22.152 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-08-24T01:17:22.152 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-08-24T01:17:22.152 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-08-24T01:17:22.152 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-08-24T01:17:22.486 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-08-24T01:17:22.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-08-24T01:17:22.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-08-24T01:17:22.487 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-08-24T01:17:22.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-08-24T01:17:22.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-08-24T01:17:22.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-08-24T01:17:22.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: flush_pg_stats 2020-08-24T01:17:22.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-08-24T01:17:22.924 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-08-24T01:17:23.270 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-08-24T01:17:23.270 INFO:tasks.workunit.client.0.smithi060.stderr:1 2020-08-24T01:17:23.270 INFO:tasks.workunit.client.0.smithi060.stderr:2 2020-08-24T01:17:23.270 INFO:tasks.workunit.client.0.smithi060.stderr:3 2020-08-24T01:17:23.271 INFO:tasks.workunit.client.0.smithi060.stderr:4 2020-08-24T01:17:23.271 INFO:tasks.workunit.client.0.smithi060.stderr:5' 2020-08-24T01:17:23.271 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-08-24T01:17:23.271 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:23.272 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-08-24T01:17:23.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836496 2020-08-24T01:17:23.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 21474836496 2020-08-24T01:17:23.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836496' 2020-08-24T01:17:23.416 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:23.417 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-08-24T01:17:23.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672975 2020-08-24T01:17:23.562 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 42949672975 2020-08-24T01:17:23.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836496 1-42949672975' 2020-08-24T01:17:23.563 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:23.563 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-08-24T01:17:23.715 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509454 2020-08-24T01:17:23.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 64424509454 2020-08-24T01:17:23.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509454' 2020-08-24T01:17:23.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:23.717 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-08-24T01:17:23.874 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=85899345932 2020-08-24T01:17:23.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 85899345932 2020-08-24T01:17:23.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509454 3-85899345932' 2020-08-24T01:17:23.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:23.875 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-08-24T01:17:24.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=107374182411 2020-08-24T01:17:24.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 107374182411 2020-08-24T01:17:24.029 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509454 3-85899345932 4-107374182411' 2020-08-24T01:17:24.030 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-08-24T01:17:24.030 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-08-24T01:17:24.182 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=128849018890 2020-08-24T01:17:24.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: test -z 128849018890 2020-08-24T01:17:24.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: seqs=' 0-21474836496 1-42949672975 2-64424509454 3-85899345932 4-107374182411 5-128849018890' 2020-08-24T01:17:24.183 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:24.184 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 0-21474836496 2020-08-24T01:17:24.184 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:24.184 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=0 2020-08-24T01:17:24.187 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 0-21474836496 2020-08-24T01:17:24.187 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:24.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=21474836496 2020-08-24T01:17:24.188 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2020-08-24T01:17:24.188 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.0 seq 21474836496 2020-08-24T01:17:24.189 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 0 2020-08-24T01:17:24.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 21474836496 -lt 21474836496 2020-08-24T01:17:24.582 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:24.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 1-42949672975 2020-08-24T01:17:24.583 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:24.584 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=1 2020-08-24T01:17:24.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 1-42949672975 2020-08-24T01:17:24.585 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:24.587 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=42949672975 2020-08-24T01:17:24.588 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.1 seq 42949672975' 2020-08-24T01:17:24.588 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.1 seq 42949672975 2020-08-24T01:17:24.588 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 1 2020-08-24T01:17:24.990 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 42949672975 -lt 42949672975 2020-08-24T01:17:24.991 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:24.991 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:24.992 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 2-64424509454 2020-08-24T01:17:24.992 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=2 2020-08-24T01:17:24.993 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 2-64424509454 2020-08-24T01:17:24.994 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:24.995 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=64424509454 2020-08-24T01:17:24.995 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.2 seq 64424509454' 2020-08-24T01:17:24.995 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.2 seq 64424509454 2020-08-24T01:17:24.996 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 2 2020-08-24T01:17:25.359 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 64424509454 -lt 64424509454 2020-08-24T01:17:25.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:25.362 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 3-85899345932 2020-08-24T01:17:25.363 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:25.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=3 2020-08-24T01:17:25.365 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 3-85899345932 2020-08-24T01:17:25.366 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:25.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=85899345932 2020-08-24T01:17:25.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.3 seq 85899345932' 2020-08-24T01:17:25.366 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 3 2020-08-24T01:17:25.367 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.3 seq 85899345932 2020-08-24T01:17:25.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 85899345932 -lt 85899345932 2020-08-24T01:17:25.751 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:25.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 4-107374182411 2020-08-24T01:17:25.751 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:25.752 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=4 2020-08-24T01:17:25.752 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 4-107374182411 2020-08-24T01:17:25.753 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:25.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=107374182411 2020-08-24T01:17:25.753 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.4 seq 107374182411' 2020-08-24T01:17:25.754 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.4 seq 107374182411 2020-08-24T01:17:25.754 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 4 2020-08-24T01:17:26.124 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 107374182411 -lt 107374182411 2020-08-24T01:17:26.125 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: for s in $seqs 2020-08-24T01:17:26.125 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: echo 5-128849018890 2020-08-24T01:17:26.126 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: cut -d - -f 1 2020-08-24T01:17:26.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: osd=5 2020-08-24T01:17:26.127 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: echo 5-128849018890 2020-08-24T01:17:26.128 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: cut -d - -f 2 2020-08-24T01:17:26.129 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: seq=128849018890 2020-08-24T01:17:26.130 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2082: flush_pg_stats: echo 'waiting osd.5 seq 128849018890' 2020-08-24T01:17:26.130 INFO:tasks.workunit.client.0.smithi060.stdout:waiting osd.5 seq 128849018890 2020-08-24T01:17:26.130 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: ceph osd last-stat-seq 5 2020-08-24T01:17:26.499 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2083: flush_pg_stats: test 128849018890 -lt 128849018890 2020-08-24T01:17:26.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: ceph pg dump pgs --format=json 2020-08-24T01:17:26.500 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: jq '.pg_stats[0].up_primary' 2020-08-24T01:17:26.861 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:17:26.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:79: _common_test: newprimary=4 2020-08-24T01:17:26.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: kill_daemons 2020-08-24T01:17:26.878 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:17:26.878 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:17:26.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:17:26.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:17:26.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:17:27.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:17:27.197 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: ERRORS=0 2020-08-24T01:17:27.198 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-08-24T01:17:27.198 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-08-24T01:17:27.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:27.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-08-24T01:17:27.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-08-24T01:17:27.199 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-08-24T01:17:27.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-08-24T01:17:27.200 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-08-24T01:17:27.755 INFO:tasks.workunit.client.0.smithi060.stdout:{ 2020-08-24T01:17:27.756 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_log_t": { 2020-08-24T01:17:27.756 INFO:tasks.workunit.client.0.smithi060.stdout: "head": "34'40", 2020-08-24T01:17:27.756 INFO:tasks.workunit.client.0.smithi060.stdout: "tail": "0'0", 2020-08-24T01:17:27.756 INFO:tasks.workunit.client.0.smithi060.stdout: "log": [ 2020-08-24T01:17:27.757 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.757 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.757 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:c00128d9:::obj-1:head", 2020-08-24T01:17:27.757 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'1", 2020-08-24T01:17:27.757 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.758 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4189.0:1", 2020-08-24T01:17:27.758 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.758 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:00.286212+0000", 2020-08-24T01:17:27.758 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.758 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.759 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.759 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.759 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.759 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.759 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.760 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.760 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.760 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.760 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.761 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.761 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.761 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.761 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.762 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.762 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.762 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.762 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:d7942fc0:::obj-2:head", 2020-08-24T01:17:27.762 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'2", 2020-08-24T01:17:27.763 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.763 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4192.0:1", 2020-08-24T01:17:27.763 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.763 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:00.558039+0000", 2020-08-24T01:17:27.764 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.764 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.764 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.764 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.764 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.765 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.765 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.765 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.765 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.766 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.766 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.766 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.766 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.766 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.767 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.767 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.769 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.771 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.772 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:1dda57ab:::obj-3:head", 2020-08-24T01:17:27.774 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'3", 2020-08-24T01:17:27.776 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.778 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4195.0:1", 2020-08-24T01:17:27.778 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.780 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:00.734525+0000", 2020-08-24T01:17:27.781 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.783 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.785 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.786 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.788 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.789 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.790 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.791 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.791 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.791 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.792 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.792 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.792 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.792 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.792 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.793 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.793 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.793 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.793 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:ba1802d0:::obj-4:head", 2020-08-24T01:17:27.793 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'4", 2020-08-24T01:17:27.794 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.794 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4198.0:1", 2020-08-24T01:17:27.794 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.794 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:00.950632+0000", 2020-08-24T01:17:27.794 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.795 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.795 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.795 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.795 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.795 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.796 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.796 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.796 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.796 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.796 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.797 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.797 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.797 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.797 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.797 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.798 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.798 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.798 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:d5d4a582:::obj-5:head", 2020-08-24T01:17:27.798 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'5", 2020-08-24T01:17:27.798 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.799 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4201.0:1", 2020-08-24T01:17:27.799 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.799 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:01.120919+0000", 2020-08-24T01:17:27.799 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.799 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.799 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.800 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.800 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.800 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.800 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.800 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.801 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.801 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.801 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.801 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.801 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.802 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.802 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.802 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.802 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.802 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.803 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:25cac09a:::obj-6:head", 2020-08-24T01:17:27.803 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'6", 2020-08-24T01:17:27.803 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.803 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4204.0:1", 2020-08-24T01:17:27.803 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.804 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:01.291075+0000", 2020-08-24T01:17:27.804 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.804 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.804 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.804 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.804 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.805 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.805 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.805 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.805 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.805 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.806 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.806 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.806 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.806 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.806 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.807 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.807 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.807 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.807 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:b11b7838:::obj-7:head", 2020-08-24T01:17:27.807 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'7", 2020-08-24T01:17:27.807 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.808 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4207.0:1", 2020-08-24T01:17:27.808 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.808 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:01.444690+0000", 2020-08-24T01:17:27.808 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.808 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.809 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.809 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.809 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.809 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.809 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.810 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.810 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.810 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.810 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.810 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.810 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.811 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.811 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.811 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.811 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.811 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.812 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:a0f702de:::obj-8:head", 2020-08-24T01:17:27.812 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'8", 2020-08-24T01:17:27.812 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.812 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4210.0:1", 2020-08-24T01:17:27.812 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.812 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:01.586612+0000", 2020-08-24T01:17:27.813 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.813 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.813 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.813 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.813 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.814 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.814 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.814 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.814 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.814 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.815 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.815 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.815 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.815 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.815 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.815 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.816 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.816 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.816 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:1e5c3677:::obj-9:head", 2020-08-24T01:17:27.816 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'9", 2020-08-24T01:17:27.816 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.816 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4213.0:1", 2020-08-24T01:17:27.817 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.817 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:01.751827+0000", 2020-08-24T01:17:27.817 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.817 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.817 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.818 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.818 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.818 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.818 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.818 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.819 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.819 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.819 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.819 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.819 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.819 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.820 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.820 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.820 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.820 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.821 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:86dc045c:::obj-10:head", 2020-08-24T01:17:27.821 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'10", 2020-08-24T01:17:27.821 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.821 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4216.0:1", 2020-08-24T01:17:27.821 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.822 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:01.914417+0000", 2020-08-24T01:17:27.822 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.822 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.822 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.822 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.822 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.823 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.823 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.823 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.823 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.823 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.824 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.824 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.824 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.824 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.824 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.825 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.825 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.825 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.825 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:1e6eca61:::obj-11:head", 2020-08-24T01:17:27.825 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'11", 2020-08-24T01:17:27.825 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.826 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4219.0:1", 2020-08-24T01:17:27.826 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.826 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:02.110613+0000", 2020-08-24T01:17:27.826 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.826 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.827 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.827 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.827 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.827 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.827 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.827 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.828 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.828 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.828 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.828 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.828 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.829 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.829 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.829 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.829 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.829 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.830 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:18dee445:::obj-12:head", 2020-08-24T01:17:27.830 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'12", 2020-08-24T01:17:27.830 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.830 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4222.0:1", 2020-08-24T01:17:27.830 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.831 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:02.260426+0000", 2020-08-24T01:17:27.831 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.831 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.831 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.831 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.831 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.832 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.832 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.832 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.832 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.832 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.833 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.833 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.833 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.833 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.833 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.834 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.834 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.834 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.834 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:283c5c90:::obj-13:head", 2020-08-24T01:17:27.834 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'13", 2020-08-24T01:17:27.834 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.835 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4225.0:1", 2020-08-24T01:17:27.835 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.835 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:02.441575+0000", 2020-08-24T01:17:27.835 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.835 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.836 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.836 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.836 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.836 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.836 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.837 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.837 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.837 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.837 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.837 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.838 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.838 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.838 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.838 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.838 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.838 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.839 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:cc896bf0:::obj-14:head", 2020-08-24T01:17:27.839 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'14", 2020-08-24T01:17:27.839 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.839 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4228.0:1", 2020-08-24T01:17:27.839 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.840 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:02.605429+0000", 2020-08-24T01:17:27.840 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.840 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.840 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.840 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.840 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.841 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.841 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.841 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.841 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.841 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.842 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.842 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.842 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.842 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.842 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.842 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.843 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.843 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.843 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:a581f807:::obj-15:head", 2020-08-24T01:17:27.843 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'15", 2020-08-24T01:17:27.843 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.844 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4231.0:1", 2020-08-24T01:17:27.844 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.844 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:02.764269+0000", 2020-08-24T01:17:27.844 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.844 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.845 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.845 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.845 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.845 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.845 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.845 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.846 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.846 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.846 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.846 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.846 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.847 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.847 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.847 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.847 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.847 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.847 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:c902ca7c:::obj-16:head", 2020-08-24T01:17:27.848 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'16", 2020-08-24T01:17:27.848 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.848 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4234.0:1", 2020-08-24T01:17:27.848 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.848 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:02.936252+0000", 2020-08-24T01:17:27.849 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.849 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.849 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.849 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.849 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.849 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.850 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.850 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.850 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.850 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.850 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.851 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.851 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.851 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.851 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.851 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.851 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.852 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.852 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:655283c8:::obj-17:head", 2020-08-24T01:17:27.852 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'17", 2020-08-24T01:17:27.852 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.852 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4237.0:1", 2020-08-24T01:17:27.853 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.853 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:03.082303+0000", 2020-08-24T01:17:27.853 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.853 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.853 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.854 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.854 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.854 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.854 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.854 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.855 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.855 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.855 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.855 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.855 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.855 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.856 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.856 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.856 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.856 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.856 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:d759d762:::obj-18:head", 2020-08-24T01:17:27.857 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'18", 2020-08-24T01:17:27.857 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.857 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4240.0:1", 2020-08-24T01:17:27.857 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.857 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:03.277117+0000", 2020-08-24T01:17:27.858 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.858 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.858 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.858 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.858 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.858 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.859 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.859 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.859 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.859 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.859 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.860 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.860 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.860 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.860 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.860 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.860 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.861 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.861 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:1b934a2d:::obj-19:head", 2020-08-24T01:17:27.861 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'19", 2020-08-24T01:17:27.861 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.861 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4243.0:1", 2020-08-24T01:17:27.862 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.862 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:03.464088+0000", 2020-08-24T01:17:27.862 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.862 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.862 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.863 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.863 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.863 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.863 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.863 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.863 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.864 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.864 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.864 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.864 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.864 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.865 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.865 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.865 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.865 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.865 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:1d0ea2ca:::obj-20:head", 2020-08-24T01:17:27.866 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'20", 2020-08-24T01:17:27.866 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.866 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4246.0:1", 2020-08-24T01:17:27.866 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.866 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:03.614350+0000", 2020-08-24T01:17:27.867 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.867 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.867 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.867 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.867 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.867 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.868 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.868 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.868 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.868 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.868 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.869 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.869 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.869 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.869 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.869 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.870 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.870 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.870 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:ab396690:::obj-21:head", 2020-08-24T01:17:27.870 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'21", 2020-08-24T01:17:27.870 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.870 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4249.0:1", 2020-08-24T01:17:27.871 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.871 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:03.790382+0000", 2020-08-24T01:17:27.871 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.871 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.871 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.872 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.872 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.872 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.872 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.872 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.873 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.873 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.873 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.873 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.873 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.874 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.874 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.874 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.874 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.874 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.874 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:254983db:::obj-22:head", 2020-08-24T01:17:27.875 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'22", 2020-08-24T01:17:27.875 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.875 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4252.0:1", 2020-08-24T01:17:27.875 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.875 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:03.954320+0000", 2020-08-24T01:17:27.876 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.876 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.876 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.876 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.876 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.877 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.877 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.877 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.877 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.877 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.878 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.878 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.878 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.878 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.878 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.879 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.879 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.879 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.879 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:e656299f:::obj-23:head", 2020-08-24T01:17:27.879 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'23", 2020-08-24T01:17:27.879 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.880 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4255.0:1", 2020-08-24T01:17:27.880 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.880 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:04.114692+0000", 2020-08-24T01:17:27.880 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.880 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.881 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.881 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.881 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.881 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.881 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.881 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.882 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.882 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.882 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.882 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.882 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.883 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.883 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.883 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.883 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.883 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.884 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:5cdf412b:::obj-24:head", 2020-08-24T01:17:27.884 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'24", 2020-08-24T01:17:27.884 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.884 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4258.0:1", 2020-08-24T01:17:27.884 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.885 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:04.276117+0000", 2020-08-24T01:17:27.885 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.885 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.885 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.885 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.885 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.886 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.886 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.886 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.886 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.886 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.887 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.887 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.887 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.887 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.887 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.888 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.888 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.888 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.888 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:cacae143:::obj-25:head", 2020-08-24T01:17:27.888 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'25", 2020-08-24T01:17:27.888 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.889 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4261.0:1", 2020-08-24T01:17:27.889 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.889 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:04.446199+0000", 2020-08-24T01:17:27.889 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.889 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.890 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.890 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.890 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.890 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.890 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.890 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.891 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.891 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.891 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.891 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.891 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.892 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.892 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.892 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.892 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.892 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.892 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:34b5bff6:::obj-26:head", 2020-08-24T01:17:27.893 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'26", 2020-08-24T01:17:27.893 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.893 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4264.0:1", 2020-08-24T01:17:27.893 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.893 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:04.613250+0000", 2020-08-24T01:17:27.894 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.894 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.894 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.894 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.894 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.894 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.895 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.895 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.895 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.895 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.895 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.896 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.896 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.896 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.896 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.896 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.896 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.897 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.897 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:a354f8f6:::obj-27:head", 2020-08-24T01:17:27.897 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'27", 2020-08-24T01:17:27.897 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.897 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4267.0:1", 2020-08-24T01:17:27.898 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.898 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:04.751838+0000", 2020-08-24T01:17:27.898 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.898 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.898 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.899 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.899 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.899 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.899 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.899 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.899 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.900 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.900 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.900 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.900 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.900 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.901 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.901 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.901 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.901 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.901 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:7421db49:::obj-28:head", 2020-08-24T01:17:27.902 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'28", 2020-08-24T01:17:27.902 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.902 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4270.0:1", 2020-08-24T01:17:27.902 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.902 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:04.902971+0000", 2020-08-24T01:17:27.902 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.903 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.903 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.903 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.903 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.903 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.904 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.904 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.904 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.904 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.904 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.904 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.905 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.905 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.905 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.905 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.905 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.906 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.906 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:54bc79ba:::obj-29:head", 2020-08-24T01:17:27.906 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'29", 2020-08-24T01:17:27.906 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.906 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4273.0:1", 2020-08-24T01:17:27.907 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.907 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:05.047946+0000", 2020-08-24T01:17:27.907 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.907 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.907 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.907 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.908 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.908 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.908 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.908 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.908 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.909 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.909 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.909 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.909 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.909 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.910 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.910 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.910 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.910 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.910 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:01b15595:::obj-30:head", 2020-08-24T01:17:27.910 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'30", 2020-08-24T01:17:27.911 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.911 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4276.0:1", 2020-08-24T01:17:27.911 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.911 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:05.208813+0000", 2020-08-24T01:17:27.911 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.912 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.912 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.912 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.912 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.912 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.913 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.913 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.913 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.913 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.913 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.913 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.914 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.914 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.914 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.914 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.914 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.915 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.915 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:f9a0a8a2:::obj-31:head", 2020-08-24T01:17:27.915 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'31", 2020-08-24T01:17:27.915 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.915 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4279.0:1", 2020-08-24T01:17:27.915 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.916 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:05.366373+0000", 2020-08-24T01:17:27.916 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.916 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.916 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.916 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.917 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.917 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.917 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.917 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.917 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.917 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.918 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.918 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.918 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.918 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.918 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.919 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.919 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.919 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.919 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:1bf2562f:::obj-32:head", 2020-08-24T01:17:27.919 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'32", 2020-08-24T01:17:27.919 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.920 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4282.0:1", 2020-08-24T01:17:27.920 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.920 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:05.525579+0000", 2020-08-24T01:17:27.920 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.920 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.921 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.921 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.921 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.921 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.921 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.922 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.922 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.922 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.922 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.922 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.922 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.923 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.923 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.923 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.923 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.923 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.924 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:b982ae22:::obj-33:head", 2020-08-24T01:17:27.924 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'33", 2020-08-24T01:17:27.924 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.924 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4285.0:1", 2020-08-24T01:17:27.924 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.924 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:05.678734+0000", 2020-08-24T01:17:27.925 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.925 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.925 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.925 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.925 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.926 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.926 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.926 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.926 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.926 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.926 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.927 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.927 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.927 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.927 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.927 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.928 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.928 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.928 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:1ad9f98e:::obj-34:head", 2020-08-24T01:17:27.928 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'34", 2020-08-24T01:17:27.928 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.928 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4288.0:1", 2020-08-24T01:17:27.929 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.929 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:05.842480+0000", 2020-08-24T01:17:27.929 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.929 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.929 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.930 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.930 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.930 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.930 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.930 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.930 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.931 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.931 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.931 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.931 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.932 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.932 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.932 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.933 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.933 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.933 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:941f43ff:::obj-35:head", 2020-08-24T01:17:27.934 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'35", 2020-08-24T01:17:27.934 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.934 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4291.0:1", 2020-08-24T01:17:27.935 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.935 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:05.986232+0000", 2020-08-24T01:17:27.935 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.936 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.936 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.936 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.936 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.937 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.937 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.937 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.938 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.944 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.944 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.945 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.945 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.945 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.945 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.945 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.945 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.946 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.946 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:24ebabe9:::obj-36:head", 2020-08-24T01:17:27.946 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'36", 2020-08-24T01:17:27.946 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:27.946 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4294.0:1", 2020-08-24T01:17:27.947 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:27.947 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:06.143778+0000", 2020-08-24T01:17:27.947 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:27.947 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:27.947 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:27.947 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:27.948 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:27.948 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:27.948 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.948 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.948 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:27.949 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:27.949 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:27.949 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:27.949 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:27.949 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.949 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:27.950 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:27.950 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:27.950 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:27.950 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:5f3a5398:::obj-37:head", 2020-08-24T01:17:27.950 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'37", 2020-08-24T01:17:27.951 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:28.298 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4297.0:1", 2020-08-24T01:17:28.299 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:28.299 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:06.296322+0000", 2020-08-24T01:17:28.299 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:28.299 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:28.300 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:28.300 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:28.300 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:28.300 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:28.300 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.300 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.301 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:28.301 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:28.301 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:28.301 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:28.301 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:28.302 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.302 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.302 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.302 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:28.302 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:28.303 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:c96cb37e:::obj-38:head", 2020-08-24T01:17:28.303 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'38", 2020-08-24T01:17:28.303 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:28.303 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4300.0:1", 2020-08-24T01:17:28.303 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:28.304 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:06.472044+0000", 2020-08-24T01:17:28.304 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:28.304 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:28.304 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:28.304 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:28.304 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:28.305 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:28.305 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.305 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.305 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:28.305 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:28.306 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:28.306 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:28.307 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:28.307 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.307 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.307 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.307 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:28.308 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:28.308 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:69e36996:::obj-39:head", 2020-08-24T01:17:28.308 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'39", 2020-08-24T01:17:28.308 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:28.308 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4303.0:1", 2020-08-24T01:17:28.308 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:28.309 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:06.625984+0000", 2020-08-24T01:17:28.309 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:28.309 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:28.309 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:28.309 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:28.310 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:28.310 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:28.310 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.310 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.310 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:28.311 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:28.311 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:28.311 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:28.311 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:28.311 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.311 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.312 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.312 INFO:tasks.workunit.client.0.smithi060.stdout: { 2020-08-24T01:17:28.312 INFO:tasks.workunit.client.0.smithi060.stdout: "op": "modify", 2020-08-24T01:17:28.312 INFO:tasks.workunit.client.0.smithi060.stdout: "object": "1:9f75773e:::obj-40:head", 2020-08-24T01:17:28.312 INFO:tasks.workunit.client.0.smithi060.stdout: "version": "34'40", 2020-08-24T01:17:28.313 INFO:tasks.workunit.client.0.smithi060.stdout: "prior_version": "0'0", 2020-08-24T01:17:28.313 INFO:tasks.workunit.client.0.smithi060.stdout: "reqid": "client.4306.0:1", 2020-08-24T01:17:28.313 INFO:tasks.workunit.client.0.smithi060.stdout: "extra_reqids": [], 2020-08-24T01:17:28.313 INFO:tasks.workunit.client.0.smithi060.stdout: "mtime": "2020-08-24T01:17:06.815403+0000", 2020-08-24T01:17:28.313 INFO:tasks.workunit.client.0.smithi060.stdout: "return_code": 0, 2020-08-24T01:17:28.316 INFO:tasks.workunit.client.0.smithi060.stdout: "mod_desc": { 2020-08-24T01:17:28.316 INFO:tasks.workunit.client.0.smithi060.stdout: "object_mod_desc": { 2020-08-24T01:17:28.316 INFO:tasks.workunit.client.0.smithi060.stdout: "can_local_rollback": false, 2020-08-24T01:17:28.316 INFO:tasks.workunit.client.0.smithi060.stdout: "rollback_info_completed": false, 2020-08-24T01:17:28.316 INFO:tasks.workunit.client.0.smithi060.stdout: "ops": [] 2020-08-24T01:17:28.317 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.317 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.317 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_regions": { 2020-08-24T01:17:28.317 INFO:tasks.workunit.client.0.smithi060.stdout: "object_clean_regions": { 2020-08-24T01:17:28.317 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_offsets": "[7589~18446744073709544026]", 2020-08-24T01:17:28.318 INFO:tasks.workunit.client.0.smithi060.stdout: "clean_omap": true, 2020-08-24T01:17:28.318 INFO:tasks.workunit.client.0.smithi060.stdout: "new_object": false 2020-08-24T01:17:28.318 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.318 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.318 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.318 INFO:tasks.workunit.client.0.smithi060.stdout: ], 2020-08-24T01:17:28.319 INFO:tasks.workunit.client.0.smithi060.stdout: "dups": [] 2020-08-24T01:17:28.319 INFO:tasks.workunit.client.0.smithi060.stdout: }, 2020-08-24T01:17:28.320 INFO:tasks.workunit.client.0.smithi060.stdout: "pg_missing_t": { 2020-08-24T01:17:28.320 INFO:tasks.workunit.client.0.smithi060.stdout: "missing": [], 2020-08-24T01:17:28.320 INFO:tasks.workunit.client.0.smithi060.stdout: "may_include_deletes": true 2020-08-24T01:17:28.320 INFO:tasks.workunit.client.0.smithi060.stdout: } 2020-08-24T01:17:28.320 INFO:tasks.workunit.client.0.smithi060.stdout:} 2020-08-24T01:17:28.322 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-08-24T01:17:28.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:84: _common_test: LOGLEN=40 2020-08-24T01:17:28.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:85: _common_test: '[' 40 '!=' 40 ']' 2020-08-24T01:17:28.322 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-08-24T01:17:28.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:89: _common_test: DUPSLEN=0 2020-08-24T01:17:28.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:90: _common_test: '[' 0 '!=' 0 ']' 2020-08-24T01:17:28.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-08-24T01:17:28.323 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after v 0'0 2020-08-24T01:17:28.323 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'40 2020-08-24T01:17:28.324 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'39 2020-08-24T01:17:28.324 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'38 2020-08-24T01:17:28.324 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'37 2020-08-24T01:17:28.324 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'36 2020-08-24T01:17:28.324 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'35 2020-08-24T01:17:28.325 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'34 2020-08-24T01:17:28.325 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'33 2020-08-24T01:17:28.325 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'32 2020-08-24T01:17:28.325 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'31 2020-08-24T01:17:28.325 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'30 2020-08-24T01:17:28.325 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'29 2020-08-24T01:17:28.326 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'28 2020-08-24T01:17:28.326 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'27 2020-08-24T01:17:28.326 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'26 2020-08-24T01:17:28.326 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'25 2020-08-24T01:17:28.326 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'24 2020-08-24T01:17:28.327 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'23 2020-08-24T01:17:28.327 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'22 2020-08-24T01:17:28.327 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'21 2020-08-24T01:17:28.327 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'20 2020-08-24T01:17:28.327 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'19 2020-08-24T01:17:28.328 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'18 2020-08-24T01:17:28.328 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'17 2020-08-24T01:17:28.328 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'16 2020-08-24T01:17:28.328 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'15 2020-08-24T01:17:28.328 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'14 2020-08-24T01:17:28.329 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'13 2020-08-24T01:17:28.329 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'12 2020-08-24T01:17:28.329 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'11 2020-08-24T01:17:28.329 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'10 2020-08-24T01:17:28.329 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'9 2020-08-24T01:17:28.330 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'8 2020-08-24T01:17:28.331 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'7 2020-08-24T01:17:28.331 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'6 2020-08-24T01:17:28.332 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'5 2020-08-24T01:17:28.332 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'4 2020-08-24T01:17:28.332 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'3 2020-08-24T01:17:28.332 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'2 2020-08-24T01:17:28.332 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_after copy log version 34'1 2020-08-24T01:17:28.333 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-08-24T01:17:08.012+0000 7fc61fa94700 20 copy_up_to/copy_after earliest_dup_version 0 2020-08-24T01:17:28.333 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after v 0'0 2020-08-24T01:17:28.333 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'40 2020-08-24T01:17:28.333 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'39 2020-08-24T01:17:28.333 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'38 2020-08-24T01:17:28.334 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'37 2020-08-24T01:17:28.334 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'36 2020-08-24T01:17:28.334 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'35 2020-08-24T01:17:28.334 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'34 2020-08-24T01:17:28.334 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'33 2020-08-24T01:17:28.335 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'32 2020-08-24T01:17:28.335 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'31 2020-08-24T01:17:28.335 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'30 2020-08-24T01:17:28.335 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'29 2020-08-24T01:17:28.335 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'28 2020-08-24T01:17:28.336 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'27 2020-08-24T01:17:28.336 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'26 2020-08-24T01:17:28.336 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'25 2020-08-24T01:17:28.336 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'24 2020-08-24T01:17:28.336 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'23 2020-08-24T01:17:28.337 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'22 2020-08-24T01:17:28.337 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'21 2020-08-24T01:17:28.337 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'20 2020-08-24T01:17:28.337 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'19 2020-08-24T01:17:28.337 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'18 2020-08-24T01:17:28.338 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'17 2020-08-24T01:17:28.338 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'16 2020-08-24T01:17:28.338 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'15 2020-08-24T01:17:28.338 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'14 2020-08-24T01:17:28.338 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'13 2020-08-24T01:17:28.339 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'12 2020-08-24T01:17:28.339 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'11 2020-08-24T01:17:28.339 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'10 2020-08-24T01:17:28.339 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'9 2020-08-24T01:17:28.339 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'8 2020-08-24T01:17:28.340 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'7 2020-08-24T01:17:28.340 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'6 2020-08-24T01:17:28.340 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'5 2020-08-24T01:17:28.340 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'4 2020-08-24T01:17:28.340 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'3 2020-08-24T01:17:28.340 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'2 2020-08-24T01:17:28.341 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'1 2020-08-24T01:17:28.341 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_up_to/copy_after earliest_dup_version 0 2020-08-24T01:17:28.341 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after v 0'0 2020-08-24T01:17:28.341 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'40 2020-08-24T01:17:28.341 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'39 2020-08-24T01:17:28.342 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'38 2020-08-24T01:17:28.342 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'37 2020-08-24T01:17:28.342 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'36 2020-08-24T01:17:28.342 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'35 2020-08-24T01:17:28.342 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'34 2020-08-24T01:17:28.343 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'33 2020-08-24T01:17:28.343 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'32 2020-08-24T01:17:28.343 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'31 2020-08-24T01:17:28.343 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'30 2020-08-24T01:17:28.343 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'29 2020-08-24T01:17:28.344 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'28 2020-08-24T01:17:28.344 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'27 2020-08-24T01:17:28.345 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'26 2020-08-24T01:17:28.345 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'25 2020-08-24T01:17:28.345 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'24 2020-08-24T01:17:28.345 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'23 2020-08-24T01:17:28.345 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'22 2020-08-24T01:17:28.346 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'21 2020-08-24T01:17:28.346 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'20 2020-08-24T01:17:28.352 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'19 2020-08-24T01:17:28.353 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'18 2020-08-24T01:17:28.353 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'17 2020-08-24T01:17:28.353 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'16 2020-08-24T01:17:28.353 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'15 2020-08-24T01:17:28.353 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'14 2020-08-24T01:17:28.354 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'13 2020-08-24T01:17:28.354 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'12 2020-08-24T01:17:28.354 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'11 2020-08-24T01:17:28.354 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'10 2020-08-24T01:17:28.354 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'9 2020-08-24T01:17:28.355 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'8 2020-08-24T01:17:28.355 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'7 2020-08-24T01:17:28.355 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'6 2020-08-24T01:17:28.355 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'5 2020-08-24T01:17:28.355 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'4 2020-08-24T01:17:28.356 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'3 2020-08-24T01:17:28.356 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'2 2020-08-24T01:17:28.356 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_after copy log version 34'1 2020-08-24T01:17:28.356 INFO:tasks.workunit.client.0.smithi060.stdout:td/osd-backfill-recovery-log/osd.4.log:2020-08-24T01:17:08.360+0000 7f2cc9053700 20 copy_up_to/copy_after earliest_dup_version 0 2020-08-24T01:17:28.356 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:95: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-08-24T01:17:28.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:96: _common_test: '[' 0 '!=' 0 ']' 2020-08-24T01:17:28.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-08-24T01:17:28.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:17:28.357 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T01:17:28.358 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:17:28.358 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:17:28.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:17:28.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:17:28.358 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:17:28.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:17:28.360 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:17:28.360 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:17:28.361 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:17:28.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:17:28.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:17:28.361 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:17:28.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:17:28.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:17:28.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:17:28.362 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:17:28.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:17:28.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:17:28.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T01:17:28.366 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:17:28.366 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:28.366 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:28.366 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T01:17:28.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:17:28.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:17:28.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-08-24T01:17:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-08-24T01:17:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-08-24T01:17:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.369 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-08-24T01:17:28.370 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T01:17:28.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T01:17:28.370 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T01:17:28.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T01:17:28.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T01:17:28.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T01:17:28.372 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T01:17:28.372 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T01:17:28.373 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T01:17:28.373 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T01:17:28.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T01:17:28.375 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T01:17:28.375 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T01:17:28.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:17:28.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T01:17:28.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T01:17:28.378 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T01:17:28.379 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T01:17:28.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-08-24T01:17:28.380 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T01:17:28.382 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T01:17:28.382 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:28.383 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:28.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T01:17:28.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T01:17:28.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T01:17:28.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-08-24T01:17:28.387 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-08-24T01:17:28.387 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:28.387 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:28.388 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.71396 2020-08-24T01:17:28.390 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-08-24T01:17:28.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-08-24T01:17:28.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-08-24T01:17:28.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-08-24T01:17:28.392 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_recovery_2 td/osd-backfill-recovery-log 2020-08-24T01:17:28.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:129: TEST_recovery_2: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:131: TEST_recovery_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 151 10 141 20 2020-08-24T01:17:28.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:28.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=151 2020-08-24T01:17:28.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=10 2020-08-24T01:17:28.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=141 2020-08-24T01:17:28.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=20 2020-08-24T01:17:28.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-08-24T01:17:28.394 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-08-24T01:17:28.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-08-24T01:17:28.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-08-24T01:17:28.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-08-24T01:17:28.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-08-24T01:17:28.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.610 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-08-24T01:17:28.611 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:17:28.611 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:17:28.612 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:17:28.612 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:28.612 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:28.612 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:28.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-08-24T01:17:28.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-08-24T01:17:28.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-08-24T01:17:28.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:17:28.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:17:28.790 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-08-24T01:17:28.791 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-08-24T01:17:28.792 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:17:28.792 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:17:28.792 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:17:28.793 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:17:28.793 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:28.793 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:28.794 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:17:28.794 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:17:28.794 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get fsid 2020-08-24T01:17:28.887 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-08-24T01:17:28.887 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-08-24T01:17:28.888 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-08-24T01:17:28.888 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-08-24T01:17:28.889 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-08-24T01:17:28.889 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-08-24T01:17:28.889 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-08-24T01:17:28.890 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-08-24T01:17:28.890 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-08-24T01:17:28.890 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:28.890 INFO:tasks.workunit.client.0.smithi060.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:28.890 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.71396/ceph-mon.a.asok 2020-08-24T01:17:28.891 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-08-24T01:17:28.891 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.71396/ceph-mon.a.asok config get mon_host 2020-08-24T01:17:28.984 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-08-24T01:17:28.984 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:28.984 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-08-24T01:17:28.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-08-24T01:17:28.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-08-24T01:17:28.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-08-24T01:17:28.985 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-08-24T01:17:29.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-08-24T01:17:29.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:17:29.491 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:17:29.492 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:17:29.492 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:29.493 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:29.493 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:29.494 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:17:29.495 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=50 --osd_max_pg_log_entries=50 --osd_pg_log_dups_tracked=60 --osd_pg_log_trim_min=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-08-24T01:17:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-08-24T01:17:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:29.530 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:29.531 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-08-24T01:17:29.531 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-08-24T01:17:29.533 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:17:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-08-24T01:17:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:29.534 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:17:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-08-24T01:17:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:17:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-08-24T01:17:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:17:29.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:17:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:17:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:17:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-08-24T01:17:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-08-24T01:17:29.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:17:29.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:29.537 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:17:29.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:17:29.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:17:29.537 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:17:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:17:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:29.541 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:29.541 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:17:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:17:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:17:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:17:29.542 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:17:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:17:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:17:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:29.543 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:17:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:17:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-08-24T01:17:29.544 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:17:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7d17b8a6-3284-4910-83c0-5387df98ae1e 2020-08-24T01:17:29.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 7d17b8a6-3284-4910-83c0-5387df98ae1e' 2020-08-24T01:17:29.545 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:17:29.545 INFO:tasks.workunit.client.0.smithi060.stdout:add osd0 7d17b8a6-3284-4910-83c0-5387df98ae1e 2020-08-24T01:17:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQApFUNf3cDxIBAA48p/GShVReENNAYjrKksKw== 2020-08-24T01:17:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQApFUNf3cDxIBAA48p/GShVReENNAYjrKksKw=="}' 2020-08-24T01:17:29.554 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7d17b8a6-3284-4910-83c0-5387df98ae1e -i td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:17:29.902 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:17:29.913 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-08-24T01:17:29.914 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQApFUNf3cDxIBAA48p/GShVReENNAYjrKksKw== --osd-uuid 7d17b8a6-3284-4910-83c0-5387df98ae1e 2020-08-24T01:17:29.947 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:29.940+0000 7f0f16ca2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:29.949 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:29.944+0000 7f0f16ca2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:29.950 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:29.944+0000 7f0f16ca2d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:29.950 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:29.944+0000 7f0f16ca2d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:17:29.951 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:29.944+0000 7f0f16ca2d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:17:29.951 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:29.944+0000 7f0f16ca2d80 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-08-24T01:17:29.987 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:29.980+0000 7f0f16ca2d80 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-08-24T01:17:30.373 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:30.368+0000 7f0f16ca2d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:17:33.069 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-08-24T01:17:33.070 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:17:33.070 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd0 key to auth repository 2020-08-24T01:17:33.071 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-08-24T01:17:33.071 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:17:33.556 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-08-24T01:17:33.557 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.0 2020-08-24T01:17:33.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:33.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:17:33.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:17:33.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:17:33.589 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:33.584+0000 7f4266daed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:33.591 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:33.584+0000 7f4266daed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:33.594 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:33.588+0000 7f4266daed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:33.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-08-24T01:17:33.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:17:33.908 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-08-24T01:17:33.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:17:33.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:17:33.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:33.909 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:17:33.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:33.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:17:33.910 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:17:34.126 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:34.120+0000 7f4266daed80 -1 Falling back to public interface 2020-08-24T01:17:34.274 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:17:34.872 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:34.868+0000 7f4266daed80 -1 osd.0 0 log_to_monitors {default=true} 2020-08-24T01:17:35.277 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:17:35.277 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:35.277 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:17:35.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:35.278 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-08-24T01:17:35.278 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:17:35.674 INFO:tasks.workunit.client.0.smithi060.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/95403,v1:127.0.0.1:6803/95403] [v2:127.0.0.1:6804/95403,v1:127.0.0.1:6805/95403] exists,up 7d17b8a6-3284-4910-83c0-5387df98ae1e 2020-08-24T01:17:35.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:17:35.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:17:35.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:17:35.675 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:17:35.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-08-24T01:17:35.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:35.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:17:35.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-08-24T01:17:35.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:17:35.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-08-24T01:17:35.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:17:35.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:17:35.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:17:35.677 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:17:35.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-08-24T01:17:35.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-08-24T01:17:35.678 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:17:35.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:35.679 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:17:35.680 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:17:35.682 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:17:35.683 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:17:35.685 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:17:35.686 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:35.686 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:35.686 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:35.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:35.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:17:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:17:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:17:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:17:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:17:35.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:17:35.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:17:35.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:35.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:17:35.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:17:35.688 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-08-24T01:17:35.689 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:17:35.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ed005677-1c14-4aa5-8081-3c5b96222fd2 2020-08-24T01:17:35.689 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 ed005677-1c14-4aa5-8081-3c5b96222fd2' 2020-08-24T01:17:35.689 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:17:35.701 INFO:tasks.workunit.client.0.smithi060.stdout:add osd1 ed005677-1c14-4aa5-8081-3c5b96222fd2 2020-08-24T01:17:35.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAvFUNffdyyKRAApK/NI+DI4CullVlYJmSfXA== 2020-08-24T01:17:35.702 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAvFUNffdyyKRAApK/NI+DI4CullVlYJmSfXA=="}' 2020-08-24T01:17:35.703 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ed005677-1c14-4aa5-8081-3c5b96222fd2 -i td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:17:36.293 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:17:36.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-08-24T01:17:36.307 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAvFUNffdyyKRAApK/NI+DI4CullVlYJmSfXA== --osd-uuid ed005677-1c14-4aa5-8081-3c5b96222fd2 2020-08-24T01:17:36.339 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.332+0000 7fa636f19d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:36.341 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.336+0000 7fa636f19d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:36.343 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.336+0000 7fa636f19d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:36.343 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.336+0000 7fa636f19d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:17:36.344 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.336+0000 7fa636f19d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:17:36.344 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.336+0000 7fa636f19d80 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-08-24T01:17:36.392 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.388+0000 7fa636f19d80 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-08-24T01:17:36.828 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:36.824+0000 7fa636f19d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:17:39.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-08-24T01:17:39.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:17:39.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-08-24T01:17:39.460 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:17:39.460 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd1 key to auth repository 2020-08-24T01:17:39.922 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-08-24T01:17:39.923 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.1 2020-08-24T01:17:39.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:39.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:17:39.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:17:39.924 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:17:39.954 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:39.948+0000 7f9de543fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:39.956 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:39.952+0000 7f9de543fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:39.957 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:39.952+0000 7f9de543fd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:40.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-08-24T01:17:40.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:17:40.279 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-08-24T01:17:40.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:17:40.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:17:40.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:40.280 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:17:40.281 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:40.281 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:17:40.281 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:17:40.496 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:40.488+0000 7f9de543fd80 -1 Falling back to public interface 2020-08-24T01:17:40.671 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:17:41.300 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:41.292+0000 7f9de543fd80 -1 osd.1 0 log_to_monitors {default=true} 2020-08-24T01:17:41.951 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:17:41.952 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:41.952 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:17:41.953 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:41.953 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-08-24T01:17:41.953 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:17:42.046 INFO:tasks.workunit.client.0.smithi060.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/95758,v1:127.0.0.1:6811/95758] [v2:127.0.0.1:6812/95758,v1:127.0.0.1:6813/95758] exists,up ed005677-1c14-4aa5-8081-3c5b96222fd2 2020-08-24T01:17:42.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:17:42.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:17:42.047 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:17:42.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:17:42.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-08-24T01:17:42.048 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:42.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:17:42.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-08-24T01:17:42.049 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:17:42.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-08-24T01:17:42.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:17:42.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:17:42.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:17:42.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:17:42.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-08-24T01:17:42.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-08-24T01:17:42.052 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:17:42.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:42.053 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:17:42.053 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:17:42.054 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:17:42.055 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:17:42.055 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:17:42.055 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:42.056 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:42.056 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:42.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:42.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:17:42.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:17:42.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:17:42.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:17:42.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:17:42.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:17:42.057 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:17:42.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:42.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:17:42.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:17:42.058 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-08-24T01:17:42.059 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:17:42.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d2b1cdba-8f69-497d-a4fb-9c104996efb6 2020-08-24T01:17:42.060 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 d2b1cdba-8f69-497d-a4fb-9c104996efb6' 2020-08-24T01:17:42.060 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:17:42.060 INFO:tasks.workunit.client.0.smithi060.stdout:add osd2 d2b1cdba-8f69-497d-a4fb-9c104996efb6 2020-08-24T01:17:42.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA2FUNfn9q6AxAAMbgBbqMmCMKwbbRpyScgQg== 2020-08-24T01:17:42.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA2FUNfn9q6AxAAMbgBbqMmCMKwbbRpyScgQg=="}' 2020-08-24T01:17:42.064 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d2b1cdba-8f69-497d-a4fb-9c104996efb6 -i td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:17:42.631 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:17:42.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-08-24T01:17:42.654 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA2FUNfn9q6AxAAMbgBbqMmCMKwbbRpyScgQg== --osd-uuid d2b1cdba-8f69-497d-a4fb-9c104996efb6 2020-08-24T01:17:42.678 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:42.672+0000 7fa9e5d78d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:42.681 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:42.676+0000 7fa9e5d78d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:42.682 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:42.676+0000 7fa9e5d78d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:42.683 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:42.676+0000 7fa9e5d78d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:17:42.683 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:42.676+0000 7fa9e5d78d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:17:42.683 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:42.676+0000 7fa9e5d78d80 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-08-24T01:17:42.720 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:42.716+0000 7fa9e5d78d80 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-08-24T01:17:43.072 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:43.068+0000 7fa9e5d78d80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:17:45.856 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-08-24T01:17:45.857 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:17:45.858 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd2 key to auth repository 2020-08-24T01:17:45.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-08-24T01:17:45.858 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:17:46.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-08-24T01:17:46.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:46.323 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.2 2020-08-24T01:17:46.324 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:17:46.324 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:17:46.324 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:17:46.348 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:46.344+0000 7fdd6c428d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:46.351 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:46.344+0000 7fdd6c428d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:46.352 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:46.348+0000 7fdd6c428d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:46.684 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-08-24T01:17:46.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:17:46.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-08-24T01:17:46.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:17:46.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:17:46.685 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:46.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:17:46.686 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:46.686 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:17:46.687 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:17:46.886 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:46.880+0000 7fdd6c428d80 -1 Falling back to public interface 2020-08-24T01:17:47.033 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:17:47.717 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:47.704+0000 7fdd6c428d80 -1 osd.2 0 log_to_monitors {default=true} 2020-08-24T01:17:48.034 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:17:48.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:48.035 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:17:48.035 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:17:48.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:48.036 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:17:48.395 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:17:49.397 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:17:49.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:49.398 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:17:49.398 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:17:49.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:49.399 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-08-24T01:17:49.778 INFO:tasks.workunit.client.0.smithi060.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/96113,v1:127.0.0.1:6819/96113] [v2:127.0.0.1:6820/96113,v1:127.0.0.1:6821/96113] exists,up d2b1cdba-8f69-497d-a4fb-9c104996efb6 2020-08-24T01:17:49.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:17:49.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:17:49.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:17:49.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:17:49.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-08-24T01:17:49.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:49.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:17:49.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-08-24T01:17:49.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:17:49.780 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-08-24T01:17:49.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:17:49.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:17:49.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:17:49.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:17:49.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-08-24T01:17:49.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-08-24T01:17:49.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:17:49.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:49.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:17:49.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:17:49.784 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:17:49.785 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:17:49.785 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:17:49.785 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:49.785 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:49.786 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:49.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:49.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:17:49.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:17:49.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:17:49.786 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:17:49.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:17:49.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:17:49.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:17:49.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:49.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:17:49.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:17:49.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-08-24T01:17:49.788 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:17:49.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7f1a8914-8d79-4ad7-9c10-f0fa37676976 2020-08-24T01:17:49.788 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 7f1a8914-8d79-4ad7-9c10-f0fa37676976' 2020-08-24T01:17:49.789 INFO:tasks.workunit.client.0.smithi060.stdout:add osd3 7f1a8914-8d79-4ad7-9c10-f0fa37676976 2020-08-24T01:17:49.789 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:17:49.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA9FUNfCPPjLxAAoY6fllHC/83m+yLdS6WBMg== 2020-08-24T01:17:49.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA9FUNfCPPjLxAAoY6fllHC/83m+yLdS6WBMg=="}' 2020-08-24T01:17:49.805 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7f1a8914-8d79-4ad7-9c10-f0fa37676976 -i td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:17:50.337 INFO:tasks.workunit.client.0.smithi060.stdout:3 2020-08-24T01:17:50.349 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-08-24T01:17:50.350 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA9FUNfCPPjLxAAoY6fllHC/83m+yLdS6WBMg== --osd-uuid 7f1a8914-8d79-4ad7-9c10-f0fa37676976 2020-08-24T01:17:50.382 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.376+0000 7fc8c9b0ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:50.385 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.380+0000 7fc8c9b0ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:50.386 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.380+0000 7fc8c9b0ed80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:50.387 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.380+0000 7fc8c9b0ed80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:17:50.387 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.380+0000 7fc8c9b0ed80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:17:50.387 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.380+0000 7fc8c9b0ed80 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-08-24T01:17:50.429 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.424+0000 7fc8c9b0ed80 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-08-24T01:17:50.894 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:50.892+0000 7fc8c9b0ed80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:17:53.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-08-24T01:17:53.544 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:17:53.546 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd3 key to auth repository 2020-08-24T01:17:53.546 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-08-24T01:17:53.547 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:17:54.019 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.3 2020-08-24T01:17:54.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-08-24T01:17:54.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:17:54.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:17:54.020 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:17:54.021 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:54.051 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:54.044+0000 7f33a013ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:54.053 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:54.048+0000 7f33a013ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:54.054 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:54.048+0000 7f33a013ad80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:54.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-08-24T01:17:54.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:17:54.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-08-24T01:17:54.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:17:54.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:17:54.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:54.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:17:54.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:54.368 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:17:54.369 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:17:54.590 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:54.584+0000 7f33a013ad80 -1 Falling back to public interface 2020-08-24T01:17:54.721 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:17:55.347 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:55.340+0000 7f33a013ad80 -1 osd.3 0 log_to_monitors {default=true} 2020-08-24T01:17:55.723 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:17:55.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:17:55.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:17:55.724 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:17:55.724 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:17:55.725 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-08-24T01:17:56.090 INFO:tasks.workunit.client.0.smithi060.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/96496,v1:127.0.0.1:6827/96496] [v2:127.0.0.1:6828/96496,v1:127.0.0.1:6829/96496] exists,up 7f1a8914-8d79-4ad7-9c10-f0fa37676976 2020-08-24T01:17:56.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:17:56.090 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:17:56.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:17:56.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:17:56.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-08-24T01:17:56.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:17:56.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:17:56.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-08-24T01:17:56.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:17:56.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-08-24T01:17:56.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:17:56.092 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:17:56.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:17:56.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:17:56.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-08-24T01:17:56.093 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-08-24T01:17:56.094 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:17:56.095 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:17:56.095 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:17:56.096 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:17:56.096 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:17:56.097 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:17:56.105 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:17:56.106 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:17:56.107 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:17:56.107 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:56.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:17:56.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:17:56.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:17:56.109 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:17:56.110 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:17:56.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:17:56.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:17:56.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:17:56.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:17:56.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:17:56.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:17:56.114 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-08-24T01:17:56.114 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:17:56.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=dc165a5b-c170-42d7-9b9a-6a77ea88f0bb 2020-08-24T01:17:56.116 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 dc165a5b-c170-42d7-9b9a-6a77ea88f0bb' 2020-08-24T01:17:56.117 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:17:56.118 INFO:tasks.workunit.client.0.smithi060.stdout:add osd4 dc165a5b-c170-42d7-9b9a-6a77ea88f0bb 2020-08-24T01:17:56.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBEFUNf+ca4BhAAlNT3uiY6+Ju8IiKJczfkiQ== 2020-08-24T01:17:56.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBEFUNf+ca4BhAAlNT3uiY6+Ju8IiKJczfkiQ=="}' 2020-08-24T01:17:56.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new dc165a5b-c170-42d7-9b9a-6a77ea88f0bb -i td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:17:56.705 INFO:tasks.workunit.client.0.smithi060.stdout:4 2020-08-24T01:17:56.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-08-24T01:17:56.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBEFUNf+ca4BhAAlNT3uiY6+Ju8IiKJczfkiQ== --osd-uuid dc165a5b-c170-42d7-9b9a-6a77ea88f0bb 2020-08-24T01:17:56.752 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:56.748+0000 7f95fd8ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:56.755 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:56.748+0000 7f95fd8ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:56.756 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:56.752+0000 7f95fd8ccd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:17:56.757 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:56.752+0000 7f95fd8ccd80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:17:56.758 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:56.752+0000 7f95fd8ccd80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:17:56.758 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:56.752+0000 7f95fd8ccd80 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-08-24T01:17:56.802 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:56.796+0000 7f95fd8ccd80 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-08-24T01:17:57.254 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:17:57.248+0000 7f95fd8ccd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:18:00.476 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-08-24T01:18:00.477 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:18:00.486 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd4 key to auth repository 2020-08-24T01:18:00.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-08-24T01:18:00.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:18:00.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-08-24T01:18:00.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:18:00.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:18:00.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:18:00.947 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:18:00.948 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.4 2020-08-24T01:18:00.977 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:00.972+0000 7f0f8552bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:00.981 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:00.976+0000 7f0f8552bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:00.982 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:00.976+0000 7f0f8552bd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:01.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-08-24T01:18:01.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:18:01.302 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-08-24T01:18:01.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:18:01.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:18:01.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:18:01.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:18:01.303 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:18:01.304 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:18:01.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:18:01.514 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:01.508+0000 7f0f8552bd80 -1 Falling back to public interface 2020-08-24T01:18:01.645 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:18:02.318 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:02.308+0000 7f0f8552bd80 -1 osd.4 0 log_to_monitors {default=true} 2020-08-24T01:18:02.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:18:02.651 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:18:02.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:18:02.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:18:02.652 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-08-24T01:18:02.652 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:18:03.103 INFO:tasks.workunit.client.0.smithi060.stdout:osd.4 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/96851,v1:127.0.0.1:6835/96851] [v2:127.0.0.1:6836/96851,v1:127.0.0.1:6837/96851] exists,up dc165a5b-c170-42d7-9b9a-6a77ea88f0bb 2020-08-24T01:18:03.104 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:18:03.104 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:18:03.104 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:18:03.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-08-24T01:18:03.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-08-24T01:18:03.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-08-24T01:18:03.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-08-24T01:18:03.105 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-08-24T01:18:03.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-08-24T01:18:03.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-08-24T01:18:03.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-08-24T01:18:03.106 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-08-24T01:18:03.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-08-24T01:18:03.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-08-24T01:18:03.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-08-24T01:18:03.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-08-24T01:18:03.107 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-08-24T01:18:03.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10' 2020-08-24T01:18:03.108 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-08-24T01:18:03.108 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-08-24T01:18:03.109 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-08-24T01:18:03.109 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-08-24T01:18:03.109 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-08-24T01:18:03.110 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T01:18:03.110 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T01:18:03.110 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:18:03.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' 2020-08-24T01:18:03.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-08-24T01:18:03.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-08-24T01:18:03.111 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-08-24T01:18:03.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-08-24T01:18:03.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-08-24T01:18:03.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-08-24T01:18:03.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-08-24T01:18:03.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:18:03.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-08-24T01:18:03.118 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-08-24T01:18:03.119 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-08-24T01:18:03.120 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-08-24T01:18:03.120 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9be902be-1325-4834-8a24-58c18da0ddf2 2020-08-24T01:18:03.121 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 9be902be-1325-4834-8a24-58c18da0ddf2' 2020-08-24T01:18:03.121 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-08-24T01:18:03.121 INFO:tasks.workunit.client.0.smithi060.stdout:add osd5 9be902be-1325-4834-8a24-58c18da0ddf2 2020-08-24T01:18:03.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBLFUNfjah+BxAAFi3kvtR40vnsGwGoenHbqA== 2020-08-24T01:18:03.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBLFUNfjah+BxAAFi3kvtR40vnsGwGoenHbqA=="}' 2020-08-24T01:18:03.128 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9be902be-1325-4834-8a24-58c18da0ddf2 -i td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:18:03.744 INFO:tasks.workunit.client.0.smithi060.stdout:5 2020-08-24T01:18:03.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-08-24T01:18:03.758 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBLFUNfjah+BxAAFi3kvtR40vnsGwGoenHbqA== --osd-uuid 9be902be-1325-4834-8a24-58c18da0ddf2 2020-08-24T01:18:03.790 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:03.784+0000 7fd3979acd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:03.792 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:03.788+0000 7fd3979acd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:03.794 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:03.788+0000 7fd3979acd80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:03.795 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:03.788+0000 7fd3979acd80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:18:03.795 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:03.788+0000 7fd3979acd80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:18:03.796 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:03.788+0000 7fd3979acd80 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-08-24T01:18:03.836 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:03.828+0000 7fd3979acd80 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-08-24T01:18:04.364 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:04.356+0000 7fd3979acd80 -1 freelist read_size_meta_from_db missing size meta in DB 2020-08-24T01:18:07.146 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-08-24T01:18:07.147 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-08-24T01:18:07.151 INFO:tasks.workunit.client.0.smithi060.stdout:adding osd5 key to auth repository 2020-08-24T01:18:07.151 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-08-24T01:18:07.152 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-08-24T01:18:07.615 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-08-24T01:18:07.615 INFO:tasks.workunit.client.0.smithi060.stdout:start osd.5 2020-08-24T01:18:07.616 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-08-24T01:18:07.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=0b0c971e-963a-4b0e-9e8a-8c868c1e3f6b --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=150 --osd_max_pg_log_entries=150 --osd_pg_log_dups_tracked=3000 --osd_pg_log_trim_min=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.71396/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-08-24T01:18:07.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-08-24T01:18:07.617 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-08-24T01:18:07.649 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:07.640+0000 7f8406106d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:07.650 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:07.644+0000 7f8406106d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:07.651 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:07.644+0000 7f8406106d80 -1 WARNING: all dangerous and experimental features are enabled. 2020-08-24T01:18:07.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-08-24T01:18:07.965 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-08-24T01:18:07.966 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-08-24T01:18:07.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-08-24T01:18:07.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-08-24T01:18:07.967 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:18:07.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-08-24T01:18:07.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:18:07.968 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:18:07.969 INFO:tasks.workunit.client.0.smithi060.stdout:0 2020-08-24T01:18:08.190 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:08.184+0000 7f8406106d80 -1 Falling back to public interface 2020-08-24T01:18:08.320 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:18:09.019 INFO:tasks.workunit.client.0.smithi060.stderr:2020-08-24T01:18:09.012+0000 7f8406106d80 -1 osd.5 0 log_to_monitors {default=true} 2020-08-24T01:18:09.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:18:09.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:18:09.322 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-08-24T01:18:09.322 INFO:tasks.workunit.client.0.smithi060.stdout:1 2020-08-24T01:18:09.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:18:09.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:18:09.779 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-08-24T01:18:10.781 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-08-24T01:18:10.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-08-24T01:18:10.782 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-08-24T01:18:10.782 INFO:tasks.workunit.client.0.smithi060.stdout:2 2020-08-24T01:18:10.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-08-24T01:18:10.783 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-08-24T01:18:11.193 INFO:tasks.workunit.client.0.smithi060.stdout:osd.5 up in weight 1 up_from 30 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/97206,v1:127.0.0.1:6843/97206] [v2:127.0.0.1:6844/97206,v1:127.0.0.1:6845/97206] exists,up 9be902be-1325-4834-8a24-58c18da0ddf2 2020-08-24T01:18:11.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-08-24T01:18:11.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-08-24T01:18:11.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-08-24T01:18:11.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-08-24T01:18:11.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-08-24T01:18:11.881 INFO:tasks.workunit.client.0.smithi060.stderr:pool 'test' already exists 2020-08-24T01:18:11.894 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-08-24T01:18:12.897 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 141 2020-08-24T01:18:12.898 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:12.899 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-08-24T01:18:13.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:13.043 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-08-24T01:18:13.194 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:13.195 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-08-24T01:18:13.334 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:13.335 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-08-24T01:18:13.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:13.501 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-08-24T01:18:13.642 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:13.643 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-08-24T01:18:13.829 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:13.830 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-08-24T01:18:14.000 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:14.001 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-08-24T01:18:14.157 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:14.158 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-08-24T01:18:14.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:14.301 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-08-24T01:18:14.487 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:14.488 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-08-24T01:18:14.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:14.631 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-08-24T01:18:14.800 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:14.800 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-08-24T01:18:14.940 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:14.941 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-08-24T01:18:15.094 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:15.095 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-08-24T01:18:15.228 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:15.229 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-08-24T01:18:15.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:15.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-08-24T01:18:15.507 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:15.508 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-08-24T01:18:15.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:15.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-08-24T01:18:15.801 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:15.802 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-08-24T01:18:15.952 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:15.953 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-08-24T01:18:16.096 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:16.097 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-08-24T01:18:16.232 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:16.233 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-08-24T01:18:16.383 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:16.384 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-08-24T01:18:16.525 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:16.526 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-08-24T01:18:16.658 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:16.659 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-08-24T01:18:16.793 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:16.794 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-08-24T01:18:16.944 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:16.945 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-08-24T01:18:17.095 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:17.096 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-08-24T01:18:17.246 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:17.247 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-08-24T01:18:17.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:17.422 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-08-24T01:18:17.558 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:17.559 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-08-24T01:18:17.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:17.718 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-08-24T01:18:17.885 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:17.886 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-08-24T01:18:18.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:18.028 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-08-24T01:18:18.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:18.170 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-08-24T01:18:18.312 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:18.313 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-08-24T01:18:18.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:18.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-08-24T01:18:18.605 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:18.606 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-08-24T01:18:18.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:18.757 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-08-24T01:18:18.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:18.882 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-08-24T01:18:19.026 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:19.027 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-08-24T01:18:19.192 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:19.193 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-08-24T01:18:19.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:19.367 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-08-24T01:18:19.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:19.502 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-08-24T01:18:19.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:19.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-08-24T01:18:19.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:19.807 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-08-24T01:18:19.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:19.946 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-08-24T01:18:20.112 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:20.113 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-08-24T01:18:20.256 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:20.257 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-08-24T01:18:20.425 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:20.426 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-08-24T01:18:20.564 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:20.565 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-08-24T01:18:20.716 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:20.717 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-08-24T01:18:20.875 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:20.876 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-08-24T01:18:21.009 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:21.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-08-24T01:18:21.168 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:21.168 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-08-24T01:18:21.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:21.311 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-08-24T01:18:21.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:21.462 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-08-24T01:18:21.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:21.609 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-08-24T01:18:21.755 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:21.756 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-08-24T01:18:21.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:21.881 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-08-24T01:18:22.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:22.023 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-08-24T01:18:22.165 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:22.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-08-24T01:18:22.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:22.315 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-08-24T01:18:22.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:22.459 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-08-24T01:18:22.593 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:22.594 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-08-24T01:18:22.727 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:22.727 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-08-24T01:18:22.869 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:22.870 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-08-24T01:18:23.062 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:23.063 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-08-24T01:18:23.221 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:23.222 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-08-24T01:18:23.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:23.363 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-08-24T01:18:23.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:23.515 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-08-24T01:18:23.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:23.691 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-08-24T01:18:23.876 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:23.877 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-08-24T01:18:24.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:24.010 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-08-24T01:18:24.159 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:24.160 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-08-24T01:18:24.304 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:24.305 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-08-24T01:18:24.454 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:24.455 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-08-24T01:18:24.590 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:24.591 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-08-24T01:18:24.719 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:24.720 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-08-24T01:18:24.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:24.861 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-08-24T01:18:25.012 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:25.012 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-08-24T01:18:25.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:25.196 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-08-24T01:18:25.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:25.331 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-08-24T01:18:25.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:25.467 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-08-24T01:18:25.613 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:25.614 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-08-24T01:18:25.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:25.750 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-08-24T01:18:25.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:25.901 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-08-24T01:18:26.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:26.083 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-08-24T01:18:26.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:26.234 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-08-24T01:18:26.376 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:26.377 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-08-24T01:18:26.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:26.519 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-08-24T01:18:26.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:26.653 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-08-24T01:18:26.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:26.787 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-08-24T01:18:26.928 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:26.929 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-08-24T01:18:27.055 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:27.056 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-08-24T01:18:27.189 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:27.190 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-08-24T01:18:27.323 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:27.324 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-08-24T01:18:27.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:27.466 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-08-24T01:18:27.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:27.624 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-08-24T01:18:27.767 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:27.768 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-08-24T01:18:27.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:27.910 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-08-24T01:18:28.068 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:28.068 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-08-24T01:18:28.211 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:28.212 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-08-24T01:18:28.361 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:28.362 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-08-24T01:18:28.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:28.520 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-08-24T01:18:28.654 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:28.655 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-08-24T01:18:28.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:28.790 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-08-24T01:18:28.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:28.923 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-08-24T01:18:29.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:29.091 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-08-24T01:18:29.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:29.255 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-08-24T01:18:29.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:29.393 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-08-24T01:18:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:29.539 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-08-24T01:18:29.693 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:29.694 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-08-24T01:18:29.844 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:29.845 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-08-24T01:18:29.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:29.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-08-24T01:18:30.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:30.123 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-08-24T01:18:30.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:30.473 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-08-24T01:18:30.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:30.607 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-08-24T01:18:30.748 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:30.749 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-08-24T01:18:30.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:30.900 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-08-24T01:18:31.050 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:31.051 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-08-24T01:18:31.218 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:31.219 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-08-24T01:18:31.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:31.371 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-08-24T01:18:31.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:31.517 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-08-24T01:18:31.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:31.676 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-08-24T01:18:31.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:31.818 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-08-24T01:18:31.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:31.961 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-08-24T01:18:32.125 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:32.126 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-08-24T01:18:32.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:32.264 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-08-24T01:18:32.414 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:32.415 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-08-24T01:18:32.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:32.550 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-08-24T01:18:32.714 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:32.714 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-08-24T01:18:32.878 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:32.879 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-08-24T01:18:33.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:33.005 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-08-24T01:18:33.166 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:33.167 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-08-24T01:18:33.326 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:33.327 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-08-24T01:18:33.483 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:33.484 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-08-24T01:18:33.646 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:33.647 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-08-24T01:18:33.796 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:33.797 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-08-24T01:18:33.933 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-08-24T01:18:34.661 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-08-24T01:18:34.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-08-24T01:18:34.662 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-08-24T01:18:34.662 INFO:tasks.workunit.client.0.smithi060.stderr:dumped pgs 2020-08-24T01:18:34.663 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-08-24T01:18:34.984 INFO:tasks.workunit.client.0.smithi060.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-08-24T01:18:34.993 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 20 '!=' 0 ']' 2020-08-24T01:18:34.994 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: seq 1 20 2020-08-24T01:18:34.995 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T01:18:34.995 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-1 /etc/passwd 2020-08-24T04:12:23.175 INFO:tasks.workunit.client.0.smithi060.stderr:Terminated 2020-08-24T04:12:23.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:1: _common_test: teardown td/osd-backfill-recovery-log 1 2020-08-24T04:12:23.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-08-24T04:12:23.198 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=1 2020-08-24T04:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-08-24T04:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-08-24T04:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-08-24T04:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-08-24T04:12:23.199 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-08-24T04:12:23.200 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-08-24T04:12:23.495 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-08-24T04:12:23.496 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-08-24T04:12:23.498 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-08-24T04:12:23.499 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-08-24T04:12:23.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-08-24T04:12:23.502 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-08-24T04:12:23.503 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-08-24T04:12:23.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T04:12:23.505 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-08-24T04:12:23.506 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-08-24T04:12:23.506 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-08-24T04:12:23.508 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-08-24T04:12:23.510 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 1 = 1 ']' 2020-08-24T04:12:23.511 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: '[' -n '' ']' 2020-08-24T04:12:23.511 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:194: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2020-08-24T04:12:23.512 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:195: teardown: mv td/osd-backfill-recovery-log/mgr.x.log td/osd-backfill-recovery-log/mon.a.log td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log /home/ubuntu/cephtest/archive/log 2020-08-24T04:12:23.514 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-08-24T04:12:23.532 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-08-24T04:12:23.533 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-08-24T04:12:23.533 INFO:tasks.workunit.client.0.smithi060.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.71396 2020-08-24T04:12:23.533 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.71396 2020-08-24T04:12:23.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-08-24T04:12:23.535 INFO:tasks.workunit.client.0.smithi060.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-08-24T04:12:23.535 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T04:12:23.536 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-2 /etc/passwd 2020-08-24T05:02:23.569 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T05:02:23.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T05:02:23.603 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-3 /etc/passwd 2020-08-24T05:52:23.601 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T05:52:23.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T05:52:23.622 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-4 /etc/passwd 2020-08-24T06:42:23.634 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T06:42:23.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T06:42:23.733 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-5 /etc/passwd 2020-08-24T07:32:23.669 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T07:32:24.601 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T07:32:24.602 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-6 /etc/passwd 2020-08-24T08:22:23.702 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T08:22:26.135 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T08:22:26.136 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-7 /etc/passwd 2020-08-24T09:12:23.737 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T09:12:23.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T09:12:23.770 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-8 /etc/passwd 2020-08-24T10:02:23.770 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T10:02:23.849 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T10:02:23.849 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-9 /etc/passwd 2020-08-24T10:52:23.802 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T10:52:23.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T10:52:23.837 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-10 /etc/passwd 2020-08-24T11:42:23.839 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T11:42:23.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T11:42:23.860 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-11 /etc/passwd 2020-08-24T12:32:23.871 INFO:tasks.workunit.client.0.smithi060.stderr:failed to fetch mon config (--no-mon-config to skip) 2020-08-24T12:32:23.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:69: _common_test: for j in $(seq 1 $moreobjects) 2020-08-24T12:32:23.988 INFO:tasks.workunit.client.0.smithi060.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:71: _common_test: rados -p test put obj-more-12 /etc/passwd 2020-08-24T12:33:37.409 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-08-24T12:33:37.415 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2020-08-24T12:33:37.417 DEBUG:teuthology.task.console_log:Killing console logger for smithi060 2020-08-24T12:33:37.417 DEBUG:teuthology.exit:Finished running handlers